Project

General

Profile

Running Jobs » History » Version 17

Gianluca Petrillo, 01/22/2016 12:51 PM

1 1 Erica Snider
{{toc}}
2 1 Erica Snider
3 1 Erica Snider
Under construction... NOT YET A SOURCE OF INFORMATION!!
4 1 Erica Snider
5 3 Gianluca Petrillo
6 1 Erica Snider
h1. Running Jobs
7 1 Erica Snider
8 9 Gianluca Petrillo
This page describes the job configuration file (often, in jargon: _FHiCL file_) and how to run a job using one.
9 1 Erica Snider
10 3 Gianluca Petrillo
It is assumed that you have previously read the [[ Quick-start guide to using and developing LArSoft code ]] and the information on [[Using LArSoft on the GPVM nodes]].
11 1 Erica Snider
12 11 Gianluca Petrillo
h2. The Job Configuration
13 1 Erica Snider
14 9 Gianluca Petrillo
Once a base release is set up, it is easy to run a job. The basic unit for running a job is the job-control script, written in the FHICL language. The FHICL language provides a simple mechanism for including parameter set configurations from different files such that many job configuration files can use the same configuration for a module or service.  
15 1 Erica Snider
16 10 Gianluca Petrillo
There is also a nice "FHiCL quick start guide":https://cdcvs.fnal.gov/redmine/attachments/16021/quick_start.pdf available for more details. 
17 2 Lynn Garren
18 3 Gianluca Petrillo
19 10 Gianluca Petrillo
h3. Key Concepts in FHiCL
20 1 Erica Snider
21 1 Erica Snider
There are a few key concepts to writing a FHICL job control script.  In order, they are
22 1 Erica Snider
23 1 Erica Snider
# Including previously defined configurations for services and modules from other files.  This is done using @#include@ statements. *Be sure you don't have any trailing space or tab characters* on the @#include@ line.
24 9 Gianluca Petrillo
# Service block, denoted by @services: { }@  This block will contain configurations for ART specific services such as the @TFileService@ and the @RandomNumberGenerator@.  It also contains the configuration of LArSoft specific services[1].
25 9 Gianluca Petrillo
# Source block, denoted by @source: { }@.  This block tells the job what kind of source to expect (@EmptyEvent@ in the case of Monte Carlo generation, @RootInput@ in the case of anything downstream of a Monte Carlo generator or reconstruction, and specific modules for data from the detector), the file name for the input source if appropriate, and how many events to process.  Both the file name and number of events to process can be specified on the command line.
26 9 Gianluca Petrillo
# Output block, denoted by @outputs: { }@  This block tells the job what kind of output to make, i.e. @RootOutput@, and what the name of the output file should be.  The output file name can be specified on the command line.  It is possible to define more than one output file if one wanted to run a job that produced different output files based on filter criteria - i.e. empty events are put in one file and events with neutrinos in them are put in another. Multiple output files can only be specified in the job configuration file, not from the command line.
27 9 Gianluca Petrillo
# Physics block, denoted by @physics: { }@  This block is where all producer, analyzer, and filter modules are configured.  Sequences of producer and filter modules to run is defined in user-named _paths_ in this block.  The list of analyzers and output modules to run is defined in a separate user-named path.  The block also defines two keyword parameters, @trigger_paths@ and @end_paths@.  @trigger_paths@ contains all producer and filter paths to run, and @end_paths@ contains the analyzer and output path.
28 3 Gianluca Petrillo
29 10 Gianluca Petrillo
Comments may be included in FHiCL configuration files using the "#" character.  The @#include@ is a keyword so that the parser knows not to ignore what comes after "#include[2]".
30 9 Gianluca Petrillo
31 9 Gianluca Petrillo
fn1. In old FHiCL files you will notice that LArSoft and in general non-art service configuration is enclosed in a @user@ block, that is now deprecated.
32 9 Gianluca Petrillo
33 9 Gianluca Petrillo
fn2. Note that the FHiCL parser can't recognize a comment on the same line as a @#include@ directive.
34 1 Erica Snider
35 3 Gianluca Petrillo
36 10 Gianluca Petrillo
h3. FHiCL Rules
37 1 Erica Snider
38 10 Gianluca Petrillo
There are a few of rules to keep in mind about FHiCL:
39 10 Gianluca Petrillo
* The value of the @process_name@ parameter may not contain underscores as the process name is used in the ROOT file branch name.  Module labels may not contain underscores either, for the same reason.
40 1 Erica Snider
* Parameter set names may not contain numbers, periods, backslashes, stars, etc. They may contain underscores.
41 3 Gianluca Petrillo
* Put the values for all string parameters in double quotes, @"..."@
42 3 Gianluca Petrillo
* Specify input vectors using @[ , , ]@, i.e. if you want a vector of doubles do @MyVector: [1.0, 3e-9, -900.]@
43 10 Gianluca Petrillo
* You pick out configurations from the @PROLOG@ section(s), usually defined in the @#include@ files, using the @ @local::@ syntax.  The value after the "@::@" is the name of the configuration specified in the @PROLOG@ (see the next bullet)
44 3 Gianluca Petrillo
* You can override the value of an included configuration. For example, imagine there is a configuration specified in a included file called @mymoduleconfig@ and it contains the value @-5@ for the parameter named @myint@. One can load the configuration and then change the value of @myint@ by doing the following:
45 3 Gianluca Petrillo
*# inside the producers block:
46 1 Erica Snider
<pre>
47 3 Gianluca Petrillo
physics: {
48 3 Gianluca Petrillo
  producers: {
49 3 Gianluca Petrillo
    # ...
50 3 Gianluca Petrillo
    mymod: @local::mymoduleconfig
51 3 Gianluca Petrillo
  }
52 3 Gianluca Petrillo
}
53 3 Gianluca Petrillo
</pre>
54 3 Gianluca Petrillo
*# out of the physics block 
55 3 Gianluca Petrillo
<pre>
56 1 Erica Snider
physics.producer.mymod.myint: 1
57 1 Erica Snider
</pre>
58 3 Gianluca Petrillo
The last value for a parameter always wins. If the second line was repeated with the value @2@ instead of @1@, the job would run with @myint@ as @2@.
59 3 Gianluca Petrillo
Also note that in the example the original content of @mymoduleconfig@ is not changed when the content of @mymod@ is.
60 3 Gianluca Petrillo
61 1 Erica Snider
62 6 Gianluca Petrillo
h3. Example configuration file: @detsimmodules.fcl@
63 6 Gianluca Petrillo
64 6 Gianluca Petrillo
An example of a file with predefined configurations for modules is in the "larsim/DetSim/detsimmodules.fcl":https://cdcvs.fnal.gov/redmine/projects/larsim/repository/revisions/develop/entry/DetSim/detsimmodules.fcl file.
65 6 Gianluca Petrillo
All the definitions are inside a prologue block.
66 6 Gianluca Petrillo
The following is taken from LArSoft @v02_03_01@ (experiments now use their specific @detsimmodule_Xxxx.fcl@ configuration file though):
67 6 Gianluca Petrillo
68 6 Gianluca Petrillo
<pre>
69 6 Gianluca Petrillo
###### All files that are parameter set definitions must contain BEGIN_PROLOG as their first line ######
70 6 Gianluca Petrillo
###### This tag tells the FHICL parser that parameter set definitions are coming                  ######
71 6 Gianluca Petrillo
BEGIN_PROLOG
72 6 Gianluca Petrillo
73 6 Gianluca Petrillo
74 6 Gianluca Petrillo
###### Generic configuration of the analyser for SimWireXxxx producer output ######
75 6 Gianluca Petrillo
standard_simwireana:
76 6 Gianluca Petrillo
{ 
77 6 Gianluca Petrillo
 module_type:       "SimWireAna"
78 6 Gianluca Petrillo
 DetSimModuleLabel: "daq"
79 6 Gianluca Petrillo
}
80 6 Gianluca Petrillo
81 6 Gianluca Petrillo
###### Configuration of the SimWire module (digitization) for ArgoNeuT ######
82 6 Gianluca Petrillo
# extra comment to test check-in
83 6 Gianluca Petrillo
argoneut_simwire:
84 6 Gianluca Petrillo
{ 
85 6 Gianluca Petrillo
 module_type:        "SimWireT962"
86 6 Gianluca Petrillo
 DriftEModuleLabel:  "largeant"
87 6 Gianluca Petrillo
 ResponseFile:       "shape-argo.root"
88 6 Gianluca Petrillo
 NoiseFact:          0.132        #Noise Scale
89 6 Gianluca Petrillo
 NoiseWidth:         62.4         #Exponential Noise width (kHz)
90 6 Gianluca Petrillo
 LowCutoff:          7.5          #Low frequency filter cutoff (kHz)
91 6 Gianluca Petrillo
 FieldBins:          75
92 6 Gianluca Petrillo
 Col3DCorrection:    2.5
93 6 Gianluca Petrillo
 Ind3DCorrection:    1.5
94 6 Gianluca Petrillo
 ColFieldRespAmp:    0.0354
95 6 Gianluca Petrillo
 IndFieldRespAmp:    0.018
96 6 Gianluca Petrillo
 ShapeTimeConst:     [ 3000., 900. ]
97 6 Gianluca Petrillo
 CompressionType:    "none"        #could also be Huffman
98 6 Gianluca Petrillo
}
99 6 Gianluca Petrillo
###### Configuration of the analyser for SimWireXxxx output for ArgoNeuT (copy of generic) ######
100 6 Gianluca Petrillo
argoneut_simwireana: @local::standard_simwireana
101 6 Gianluca Petrillo
102 6 Gianluca Petrillo
###### Configuration of the SimWire module (digitization) for MicroBooNE ######
103 6 Gianluca Petrillo
microboone_simwire:  
104 6 Gianluca Petrillo
{ 
105 6 Gianluca Petrillo
 module_type:        "SimWireMicroBooNE"
106 6 Gianluca Petrillo
 DriftEModuleLabel:  "largeant"
107 6 Gianluca Petrillo
 NoiseFact:          0.0132       #Noise Scale
108 6 Gianluca Petrillo
 #NoiseFact:          0.15       #Noise Scale to use with histogram
109 6 Gianluca Petrillo
 NoiseWidth:         62.4         #Exponential Noise width (kHz)
110 6 Gianluca Petrillo
 LowCutoff:          7.5          #Low frequency filter cutoff (kHz)
111 6 Gianluca Petrillo
 CompressionType:    "none"       #could also be none
112 6 Gianluca Petrillo
 GetNoiseFromHisto:   false  
113 6 Gianluca Petrillo
 NoiseFileFname:      "uboone_noise_v0.1.root"
114 6 Gianluca Petrillo
 NoiseHistoName:      "NoiseFreq"    
115 6 Gianluca Petrillo
116 6 Gianluca Petrillo
}
117 6 Gianluca Petrillo
###### Configuration of the analyser for SimWireXxxx output for MicroBooNE (copy of generic) ######
118 6 Gianluca Petrillo
microboone_simwireana: @local::standard_simwireana
119 6 Gianluca Petrillo
120 6 Gianluca Petrillo
121 6 Gianluca Petrillo
122 6 Gianluca Petrillo
123 6 Gianluca Petrillo
124 6 Gianluca Petrillo
###### Configuration of a different analyser for SimWireXxxx output, generic and experiment-specific ######
125 6 Gianluca Petrillo
standard_wienerfilterana: 
126 6 Gianluca Petrillo
{
127 6 Gianluca Petrillo
 module_type:        "WienerFilterAna"
128 6 Gianluca Petrillo
 DetSimModuleLabel:  "daq"
129 6 Gianluca Petrillo
}
130 6 Gianluca Petrillo
131 6 Gianluca Petrillo
bo_wienerfilterana:         @local::standard_wienerfilterana
132 6 Gianluca Petrillo
argoneut_wienerfilterana:   @local::standard_wienerfilterana
133 6 Gianluca Petrillo
microboone_wienerfilterana: @local::standard_wienerfilterana
134 6 Gianluca Petrillo
135 6 Gianluca Petrillo
136 6 Gianluca Petrillo
###### All files that are parameter set definitions must contain END_PROLOG as their last line ######
137 6 Gianluca Petrillo
###### This tag tells the FHICL parser that parameter set definitions are ended                ######
138 6 Gianluca Petrillo
END_PROLOG
139 6 Gianluca Petrillo
</pre>
140 6 Gianluca Petrillo
141 5 Gianluca Petrillo
h3. Configuring the "message service":https://cdcvs.fnal.gov/redmine/projects/messagefacility/wiki/Using_MessageFacility#Using-MessageFacility
142 1 Erica Snider
143 4 Gianluca Petrillo
Several standard configurations for the message service are in "lardata/Utilities/messageservice.fcl":https://cdcvs.fnal.gov/redmine/projects/lardata/repository/revisions/develop/entry/Utilities/messageservice.fcl.  There is one configuration for each level of message output - Debug, Info, Warning, and Error.  These configurations will be applied to each message level that is specified and those of higher priority.  For example, the Info configuration will print out Info, Warning and Error level messages while the Warning configuration only print outs Warning and Error level messages. The "standard" debug configurations will cause the messages to go to a specified output file, @debug.log@. The Error configuration redirect to standard error stream (like @std::cerr@), while the others print into the standard output (@std::cout@). All impose some limits on the repetition of some frequent messages.
144 5 Gianluca Petrillo
Remember that to use one of these "standard" configurations you need to include it in your FCL file: they are standard, not default.
145 4 Gianluca Petrillo
If you want to define your own configuration, please take a look at the comments in "lardata/Utilities/messageservice.fcl":https://cdcvs.fnal.gov/redmine/projects/lardata/repository/revisions/develop/entry/Utilities/messageservice.fcl file to determine how to do so.  
146 1 Erica Snider
147 1 Erica Snider
Examples of how to include the usual use of the message service configurations are in the example files below.
148 1 Erica Snider
149 5 Gianluca Petrillo
To get a different level of output from just one module (say @DBSCAN@) one would do:
150 1 Erica Snider
151 1 Erica Snider
<pre>
152 1 Erica Snider
services:
153 1 Erica Snider
{
154 1 Erica Snider
  # Load the service that manages root files for histograms.
155 1 Erica Snider
  TFileService: { fileName: "reco_hist.root" }
156 1 Erica Snider
  Timing:       {}
157 1 Erica Snider
  RandomNumberGenerator: {} #ART native random number generator
158 1 Erica Snider
159 1 Erica Snider
  # configure the message service with the INFO for DBSCAN
160 1 Erica Snider
  # and WARNING level for everything else
161 1 Erica Snider
  message: { 
162 1 Erica Snider
    destinations: {  
163 1 Erica Snider
      infomsg: {
164 1 Erica Snider
        type: "cout"
165 1 Erica Snider
        threshold: "INFO"
166 1 Erica Snider
        append: true
167 1 Erica Snider
        category: {
168 1 Erica Snider
           DBSCAN: {
169 1 Erica Snider
             reportEvery: 1
170 1 Erica Snider
           }
171 1 Erica Snider
        }
172 1 Erica Snider
      }
173 1 Erica Snider
      warningmsg: {
174 1 Erica Snider
        type:      "cout"	
175 1 Erica Snider
        threshold: "WARNING"
176 1 Erica Snider
        append:    true        
177 1 Erica Snider
        categories: {
178 1 Erica Snider
          default: {
179 1 Erica Snider
            limit:       1000   
180 1 Erica Snider
            timespan:    60    
181 1 Erica Snider
          }
182 1 Erica Snider
        } # end categories
183 1 Erica Snider
      } # end warningmsg
184 1 Erica Snider
    } # end destinations
185 1 Erica Snider
  } # end standard_warning
186 1 Erica Snider
187 1 Erica Snider
  user:         @local::argoneut_services 	     
188 1 Erica Snider
}
189 1 Erica Snider
</pre>
190 1 Erica Snider
191 12 Gianluca Petrillo
h4. Debug messages
192 12 Gianluca Petrillo
193 12 Gianluca Petrillo
Note that debugging messages are treated in a special way:
194 12 Gianluca Petrillo
# there are two ways to print a debug message in the code: by using @mf::LogDebug@, and by using @LOG_DEBUG@ macro; the messages using the latter will not be present when a non-debug qualifier (@prof@ and @opt@) is used. In fact, no code is generated at all from a @LOG_DEBUG@ call unless using a @debug@ qualified build
195 12 Gianluca Petrillo
# debug output from modules is selectively enabled by the @debugModules@ list; to enable all the messages, use @debugModules: [ "*" ]@
196 12 Gianluca Petrillo
197 12 Gianluca Petrillo
Also remember that, depending on the configuration, the debug output could be not shown on screen, but only stored in a file (e.g. @debug.log@).
198 1 Erica Snider
199 5 Gianluca Petrillo
h3. Example job script: @prodgenie.fcl@
200 1 Erica Snider
201 5 Gianluca Petrillo
An example job script to produce Monte Carlo events is "larsim/EventGenerator/GENIE/prodgenie.fcl":https://cdcvs.fnal.gov/redmine/projects/larsim/repository/revisions/develop/entry/EventGenerator/GENIE/prodgenie.fcl .  The job defined by this script will generate neutrino interactions using GENIE, run them through Geant4, do the electron transport and then simulate the electronics.
202 1 Erica Snider
203 1 Erica Snider
Comments on the form of the file are included as ###### Commment ######
204 1 Erica Snider
205 1 Erica Snider
<pre>
206 1 Erica Snider
###### This is how to include configurations from other files ######
207 5 Gianluca Petrillo
#include "services.fcl"
208 5 Gianluca Petrillo
#include "genie.fcl"
209 5 Gianluca Petrillo
#include "largeantmodules.fcl"
210 5 Gianluca Petrillo
#include "detsimmodules.fcl"
211 1 Erica Snider
212 1 Erica Snider
###### give the process a name ######
213 1 Erica Snider
process_name: GenieGen
214 1 Erica Snider
215 5 Gianluca Petrillo
###### Please note the convention of defining detector specific configurations                ######
216 1 Erica Snider
###### Pick out the configurations from the #include files using the @local:: syntax          ######
217 1 Erica Snider
###### for services from LArSoft, in the user{} block - see definitions for configurations in ######
218 1 Erica Snider
###### job/geometry.fcl                                                                       ######
219 1 Erica Snider
###### job/services.fcl                                                                       ######
220 5 Gianluca Petrillo
###### job/simulationservices.fcl                                                             ######
221 5 Gianluca Petrillo
222 5 Gianluca Petrillo
services:
223 1 Erica Snider
{
224 5 Gianluca Petrillo
  # Load the service that manages root files for histograms.
225 1 Erica Snider
  TFileService: { fileName: "genie_hist.root" }
226 1 Erica Snider
  Timing:       {}
227 5 Gianluca Petrillo
  SimpleMemoryCheck:     { ignoreTotal: 1 } # default is one
228 5 Gianluca Petrillo
  RandomNumberGenerator: {} #ART native random number generator
229 5 Gianluca Petrillo
  user:         @local::argoneut_simulation_services
230 1 Erica Snider
}
231 1 Erica Snider
232 1 Erica Snider
###### source is where you get events from - can also be RootInput ######
233 1 Erica Snider
#Start each new event with an empty event.
234 1 Erica Snider
source:
235 1 Erica Snider
{
236 1 Erica Snider
  module_type: EmptyEvent
237 1 Erica Snider
  maxEvents:   10          # Number of events to create
238 1 Erica Snider
  firstRun:    1           # Run number to use for this file
239 1 Erica Snider
  firstEvent:  1           # number of first event in the file
240 5 Gianluca Petrillo
}
241 1 Erica Snider
242 1 Erica Snider
###### physics is the block that controls configuration of modules ######
243 5 Gianluca Petrillo
# Define and configure some modules to do work on each event.
244 1 Erica Snider
# First modules are defined; they are scheduled later.
245 5 Gianluca Petrillo
# Modules are grouped by type.
246 5 Gianluca Petrillo
physics:
247 1 Erica Snider
{
248 1 Erica Snider
249 1 Erica Snider
 ###### the module labels in the output file will be generator, largeant, and daq ######
250 1 Erica Snider
 ###### their configuration is taken from ArgoNeuT defaults                       ######
251 1 Erica Snider
 producers:
252 1 Erica Snider
 {
253 1 Erica Snider
   generator: @local::argoneut_genie_simple_neutrino
254 5 Gianluca Petrillo
   largeant:  @local::argoneut_largeant	       
255 1 Erica Snider
   daq:       @local::argoneut_simwire         
256 1 Erica Snider
   rns:       { module_type: "RandomNumberSaver" }
257 1 Erica Snider
 }
258 1 Erica Snider
259 1 Erica Snider
 #define the producer and filter modules for this path, order matters, 
260 1 Erica Snider
 #filters reject all following items.  see lines starting physics.producers below
261 1 Erica Snider
 simulate: [ rns, generator, largeant, daq, ] 
262 1 Erica Snider
 
263 1 Erica Snider
 #define the output stream, there could be more than one if using filters 
264 1 Erica Snider
 stream1:  [ out1 ]
265 1 Erica Snider
266 1 Erica Snider
 #trigger_paths is a keyword and contains the paths that modify the art::event, 
267 1 Erica Snider
 #ie filters and producers
268 1 Erica Snider
 trigger_paths: [simulate] 
269 1 Erica Snider
270 5 Gianluca Petrillo
 #end_paths is a keyword and contains the paths that do not modify the art::Event, 
271 1 Erica Snider
 #ie analyzers and output streams.  these all run simultaneously
272 5 Gianluca Petrillo
 end_paths:     [stream1]  
273 5 Gianluca Petrillo
}
274 1 Erica Snider
275 1 Erica Snider
#block to define where the output goes.  if you defined a filter in the physics
276 1 Erica Snider
#block and put it in the trigger_paths then you need to put a SelectEvents: {SelectEvents: [XXX]}
277 1 Erica Snider
#entry in the output stream you want those to go to, where XXX is the label of the filter module(s)
278 1 Erica Snider
outputs:
279 1 Erica Snider
{
280 1 Erica Snider
 out1:
281 1 Erica Snider
 {
282 1 Erica Snider
   module_type: RootOutput
283 1 Erica Snider
   fileName:    "genie_gen.root" #default file name, can override from command line with -o or --output
284 1 Erica Snider
 }
285 1 Erica Snider
}
286 1 Erica Snider
287 1 Erica Snider
</pre>
288 1 Erica Snider
289 1 Erica Snider
Notice that you have not specified which libraries to load anywhere. That is because the build system compiles the plugin shared library files (@.so@) against the ones they depend upon.  
290 1 Erica Snider
291 1 Erica Snider
292 1 Erica Snider
h3. Example job script: @standard_reco.fcl@
293 1 Erica Snider
294 6 Gianluca Petrillo
There is an example reconstruction job script available for people to use, "lardata/Utilities/standard_reco":https://cdcvs.fnal.gov/redmine/projects/lardata/repository/revisions/develop/entry/Utilities/standard_reco.fcl .
295 6 Gianluca Petrillo
This script takes the output of either raw data or MC that has produced simulated raw digits and performs a list of reconstruction tasks.
296 6 Gianluca Petrillo
The version from LArSoft @v02_03_01@ is copied here:
297 1 Erica Snider
298 1 Erica Snider
<pre>
299 6 Gianluca Petrillo
#include "services.fcl"
300 6 Gianluca Petrillo
#include "caldata.fcl"
301 6 Gianluca Petrillo
#include "hitfindermodules.fcl"
302 6 Gianluca Petrillo
#include "clustermodules.fcl"
303 6 Gianluca Petrillo
#include "trackfindermodules.fcl"
304 1 Erica Snider
305 1 Erica Snider
process_name: Reco
306 1 Erica Snider
307 1 Erica Snider
services:
308 1 Erica Snider
{
309 1 Erica Snider
  # Load the service that manages root files for histograms.
310 1 Erica Snider
  TFileService: { fileName: "reco_hist.root" }
311 1 Erica Snider
  Timing:       {}
312 1 Erica Snider
  RandomNumberGenerator: {} #ART native random number generator
313 1 Erica Snider
  message:      @local::standard_warning
314 6 Gianluca Petrillo
  user:         @local::argoneut_services 	     
315 1 Erica Snider
}
316 1 Erica Snider
317 6 Gianluca Petrillo
318 1 Erica Snider
#source is now a root file
319 1 Erica Snider
source:
320 1 Erica Snider
{
321 1 Erica Snider
  module_type: RootInput
322 1 Erica Snider
  maxEvents:  10        # Number of events to create
323 1 Erica Snider
}
324 1 Erica Snider
325 1 Erica Snider
# Define and configure some modules to do work on each event.
326 1 Erica Snider
# First modules are defined; they are scheduled later.
327 1 Erica Snider
# Modules are grouped by type.
328 1 Erica Snider
physics:
329 1 Erica Snider
{
330 1 Erica Snider
331 1 Erica Snider
 producers:
332 1 Erica Snider
 {
333 6 Gianluca Petrillo
  caldata:    @local::argoneut_calwire	    
334 6 Gianluca Petrillo
  ffthit:     @local::argoneut_hitfinder # note if job is for MC, use argoneut_mc_hitfinder
335 6 Gianluca Petrillo
  dbcluster:  @local::argoneut_dbcluster	    
336 1 Erica Snider
  hough:      @local::argoneut_houghlinefinder
337 6 Gianluca Petrillo
  linemerger: @local::argoneut_linemerger	    
338 6 Gianluca Petrillo
  track:      @local::argoneut_track	    
339 6 Gianluca Petrillo
  harris:     @local::argoneut_endpointmod    
340 1 Erica Snider
 }
341 1 Erica Snider
342 6 Gianluca Petrillo
 analyzers:
343 6 Gianluca Petrillo
 {
344 6 Gianluca Petrillo
  dbclusterana: @local::argoneut_dbclusterana
345 6 Gianluca Petrillo
 }
346 6 Gianluca Petrillo
347 1 Erica Snider
 #define the producer and filter modules for this path, order matters, 
348 1 Erica Snider
 #filters reject all following items.  see lines starting physics.producers below
349 6 Gianluca Petrillo
 reco: [ caldata, ffthit, dbcluster, hough, linemerger, track, harris ] 
350 6 Gianluca Petrillo
 ana:  [ dbclusterana ]
351 6 Gianluca Petrillo
352 1 Erica Snider
 #define the output stream, there could be more than one if using filters 
353 1 Erica Snider
 stream1:  [ out1 ]
354 1 Erica Snider
355 1 Erica Snider
 #trigger_paths is a keyword and contains the paths that modify the art::event, 
356 1 Erica Snider
 #ie filters and producers
357 1 Erica Snider
 trigger_paths: [reco] 
358 1 Erica Snider
359 1 Erica Snider
 #end_paths is a keyword and contains the paths that do not modify the art::Event, 
360 1 Erica Snider
 #ie analyzers and output streams.  these all run simultaneously
361 6 Gianluca Petrillo
 end_paths:     [ ana, stream1]  
362 1 Erica Snider
}
363 1 Erica Snider
364 1 Erica Snider
#block to define where the output goes.  if you defined a filter in the physics
365 1 Erica Snider
#block and put it in the trigger_paths then you need to put a SelectEvents: {SelectEvents: [XXX]}
366 1 Erica Snider
#entry in the output stream you want those to go to, where XXX is the label of the filter module(s)
367 1 Erica Snider
outputs:
368 1 Erica Snider
{
369 1 Erica Snider
 out1:
370 1 Erica Snider
 {
371 1 Erica Snider
   module_type: RootOutput
372 1 Erica Snider
   fileName:    "standard_reco.root" #default file name, can override from command line with -o or --output
373 1 Erica Snider
 }
374 1 Erica Snider
}
375 1 Erica Snider
376 1 Erica Snider
</pre>
377 1 Erica Snider
378 1 Erica Snider
h3. How to override a default parameter
379 1 Erica Snider
380 1 Erica Snider
If you want to override a default parameter that has been included from a predefined parameter set, you must specify which parameter and its value as
381 1 Erica Snider
382 7 Gianluca Petrillo
<pre>
383 1 Erica Snider
mainBlock.subBlock.label.parameterName: newValue
384 7 Gianluca Petrillo
</pre>
385 1 Erica Snider
386 1 Erica Snider
where 
387 1 Erica Snider
388 13 Gianluca Petrillo
* @mainBlock@ can be @services@ or @physics@ 
389 13 Gianluca Petrillo
* @subBlock@ can be @producers@, @filters@, or @analyzers@, skipped in case of services, and @user@ for old service configuration
390 13 Gianluca Petrillo
* @label@ is the name of the desired service or module instance in a producers, filters, or analyzers block
391 1 Erica Snider
* @parameterName@ is the name of the desired parameter
392 13 Gianluca Petrillo
* @newValue@ is the desired new value; a list or entire block (that is, a brace-enclosed sequence of @key: value@ pairs) can be specified
393 1 Erica Snider
394 1 Erica Snider
These lines must go after the @mainBlock@ and be outside of any other block.
395 1 Erica Snider
396 13 Gianluca Petrillo
For example, if one wanted to change the default value of the @HitsModuleLabel@ parameter in the @dbcluster@ module instance in the previous section, one would put
397 1 Erica Snider
<pre>
398 13 Gianluca Petrillo
physics.producers.dbcluster.HitsModuleLabel: "differentHitModuleLabel"
399 1 Erica Snider
</pre>
400 13 Gianluca Petrillo
Note that FHiCL allows for completely replacing a list but not for changing its size (i.e., it's not possible to add an element to an existing list).
401 1 Erica Snider
402 13 Gianluca Petrillo
This behavior is general: FHiCL only allows to replace elements. For example, if one wants to change both @eps@ and @epstwo@ parameters,
403 13 Gianluca Petrillo
<pre>
404 13 Gianluca Petrillo
physics.producers.dbcluster.DBScanAlg: { # WRONG!!!
405 13 Gianluca Petrillo
  eps:    1.1
406 13 Gianluca Petrillo
  epstwo: 1.6
407 13 Gianluca Petrillo
}
408 14 Gianluca Petrillo
</pre> will not work, because it is replacing the whole @physics.producers.dbcluster.DBScanAlg@ parameter set, thus removing e.g. the parameter @minPts@.
409 13 Gianluca Petrillo
The correct way to change parameters is to think of that as _replacing_ them, and to point exactly at each of them:
410 13 Gianluca Petrillo
<pre>
411 1 Erica Snider
physics.producers.dbcluster.DBScanAlg.eps:    1.1
412 13 Gianluca Petrillo
physics.producers.dbcluster.DBScanAlg.epstwo: 1.6
413 14 Gianluca Petrillo
</pre> will leave the other parameters (e.g., @minPts@) unchanged.
414 7 Gianluca Petrillo
415 1 Erica Snider
h3. fhicl Emacs syntax highlighting
416 1 Erica Snider
417 7 Gianluca Petrillo
If you use Emacs as your editor, you can put the following into your @.emacs@ file in your home directory to cause it to display @.fcl@ files with syntax highlighting
418 1 Erica Snider
419 1 Erica Snider
<pre>
420 1 Erica Snider
(setq fclKeywords
421 1 Erica Snider
'(
422 1 Erica Snider
  ;; This, due to poor language design, conflicts with comments and fails
423 1 Erica Snider
  ("#include" . font-lock-keyword-face)
424 1 Erica Snider
  ("@local" . font-lock-keyword-face)
425 1 Erica Snider
  ;; All these names are magic, I think
426 1 Erica Snider
427 1 Erica Snider
428 1 Erica Snider
("process_name:\\|services:\\|source:\\|outputs:\\|physics\\|producers:\\|filters:\\|analyzers:" . font-lock-builtin-face)
429 1 Erica Snider
  ("true\\|false" . font-lock-builtin-face)
430 1 Erica Snider
  ;; Variable definitions are followed by colons
431 1 Erica Snider
432 1 Erica Snider
  ("[a-zA-Z0-9_]*:" . font-lock-variable-name-face)
433 1 Erica Snider
 )
434 1 Erica Snider
)
435 1 Erica Snider
436 1 Erica Snider
;; Python mode gets us comment handling and indentation at colons
437 1 Erica Snider
438 1 Erica Snider
(define-derived-mode fcl-mode python-mode
439 1 Erica Snider
 (setq mode-name "FHICL")
440 1 Erica Snider
 (setq font-lock-defaults '(fclKeywords))
441 1 Erica Snider
;;  (setq tab-width 2) ;; Doesn't seem to work
442 1 Erica Snider
443 1 Erica Snider
)
444 1 Erica Snider
445 1 Erica Snider
(add-to-list 'auto-mode-alist '("\\.fcl\\'" . fcl-mode))
446 1 Erica Snider
</pre>
447 1 Erica Snider
448 15 Gianluca Petrillo
bq. Please use _only spaces_ for alignment: it's not the perfect solution, but it makes people using different editors see the same code alignment.
449 7 Gianluca Petrillo
450 1 Erica Snider
h2. Executable and command line options
451 1 Erica Snider
452 7 Gianluca Petrillo
Currently there is one executable to run in LArSoft. The executable to run a typical reconstruction or analysis job is @lar@ which is placed in the user's path by the setup script. To see what options are available do
453 7 Gianluca Petrillo
<pre>
454 7 Gianluca Petrillo
$ lar -h
455 7 Gianluca Petrillo
</pre>
456 16 Gianluca Petrillo
The output for @art@ version @v1_17_05@ is:
457 7 Gianluca Petrillo
<pre>
458 7 Gianluca Petrillo
Usage: lar <-c <config-file>> <other-options> [<source-file>]+
459 1 Erica Snider
460 1 Erica Snider
Allowed options:
461 16 Gianluca Petrillo
  -c [ --config ] arg                  Configuration file.
462 16 Gianluca Petrillo
  -h [ --help ]                        produce help message
463 16 Gianluca Petrillo
  --process-name arg                   art process name.
464 16 Gianluca Petrillo
  --print-available arg                List all available plugins with the
465 16 Gianluca Petrillo
                                       provided suffix.  Choose from:
466 16 Gianluca Petrillo
                                           'module'
467 16 Gianluca Petrillo
                                           'plugin'
468 16 Gianluca Petrillo
                                           'service'
469 16 Gianluca Petrillo
                                           'source'
470 16 Gianluca Petrillo
  --print-available-modules            List all available modules that can be
471 16 Gianluca Petrillo
                                       invoked in a FHiCL file.
472 16 Gianluca Petrillo
  --print-available-services           List all available services that can be
473 16 Gianluca Petrillo
                                       invoked in a FHiCL file.
474 16 Gianluca Petrillo
  --print-description arg              Print description of specified module,
475 16 Gianluca Petrillo
                                       service, source, or other plugin
476 16 Gianluca Petrillo
                                       (multiple OK).
477 1 Erica Snider
478 16 Gianluca Petrillo
  -s [ --source ] arg                  Source data file (multiple OK);
479 16 Gianluca Petrillo
                                       precludes -S.
480 16 Gianluca Petrillo
  -S [ --source-list ] arg             file containing a list of source files
481 16 Gianluca Petrillo
                                       to read, one per line; precludes -s.
482 16 Gianluca Petrillo
  -e [ --estart ] arg                  Event # of first event to process.
483 16 Gianluca Petrillo
  -n [ --nevts ] arg                   Number of events to process.
484 16 Gianluca Petrillo
  --nskip arg                          Number of events to skip.
485 16 Gianluca Petrillo
  -T [ --TFileName ] arg               File name for TFileService.
486 16 Gianluca Petrillo
  --tmpdir arg                         Temporary directory for in-progress
487 16 Gianluca Petrillo
                                       output files (defaults to directory of
488 16 Gianluca Petrillo
                                       specified output file names).
489 16 Gianluca Petrillo
  -o [ --output ] arg                  Event output stream file (optionally
490 16 Gianluca Petrillo
                                       specify stream with stream-label:fileNam
491 16 Gianluca Petrillo
                                       e in which case multiples are OK).
492 16 Gianluca Petrillo
  --no-output                          Disable all output streams.
493 16 Gianluca Petrillo
  --trace                              Activate tracing.
494 16 Gianluca Petrillo
  --notrace                            Deactivate tracing.
495 16 Gianluca Petrillo
  --memcheck                           Activate monitoring of memory use.
496 16 Gianluca Petrillo
  --memcheck-db arg                    Output memory use data to SQLite3
497 16 Gianluca Petrillo
                                       database with name <db-file>.
498 16 Gianluca Petrillo
  --nomemcheck                         Deactivate monitoring of memory use.
499 16 Gianluca Petrillo
  --default-exceptions                 Some exceptions may be handled
500 16 Gianluca Petrillo
                                       differently by default (e.g.
501 16 Gianluca Petrillo
                                       ProductNotFound).
502 16 Gianluca Petrillo
  --rethrow-default                    All exceptions default to rethrow.
503 16 Gianluca Petrillo
  --rethrow-all                        All exceptions overridden to rethrow (cf
504 16 Gianluca Petrillo
                                       rethrow-default).
505 16 Gianluca Petrillo
  --errorOnFailureToPut [=arg(=true)]  Global flag that controls the behavior
506 16 Gianluca Petrillo
                                       upon failure to 'put' a product
507 16 Gianluca Petrillo
                                       (declared by 'produces') onto the Event.
508 16 Gianluca Petrillo
                                         If 'true', per-module flags can
509 16 Gianluca Petrillo
                                       override the value of the global flag.
510 16 Gianluca Petrillo
  --errorOnSIGINT [=arg(=true)]        If 'true', a signal received from the
511 16 Gianluca Petrillo
                                       user yields an art return code
512 16 Gianluca Petrillo
                                       corresponding to an error; otherwise
513 16 Gianluca Petrillo
                                       return 0.
514 16 Gianluca Petrillo
  --debug-config arg                   Output post-processed configuration to
515 16 Gianluca Petrillo
                                       <file> and exit. Equivalent to env
516 16 Gianluca Petrillo
                                       ART_DEBUG_CONFIG=<file> lar ...
517 16 Gianluca Petrillo
  --config-out arg                     Output post-processed configuration to
518 16 Gianluca Petrillo
                                       <file> and continue with job.
519 16 Gianluca Petrillo
  --annotate                           Include configuration parameter source
520 16 Gianluca Petrillo
                                       information.
521 16 Gianluca Petrillo
  --prefix-annotate                    Include configuration parameter source
522 16 Gianluca Petrillo
                                       information on line preceding parameter
523 16 Gianluca Petrillo
                                       declaration.
524 16 Gianluca Petrillo
  --sam-web-uri arg                    URI for SAM web service.
525 16 Gianluca Petrillo
  --sam-process-id arg                 SAM process ID.
526 16 Gianluca Petrillo
  --sam-application-family arg         SAM application family.
527 16 Gianluca Petrillo
  --sam-app-family arg                 SAM application family.
528 16 Gianluca Petrillo
  --sam-application-version arg        SAM application version.
529 16 Gianluca Petrillo
  --sam-app-version arg                SAM application version.
530 16 Gianluca Petrillo
  --sam-group arg                      SAM group.
531 16 Gianluca Petrillo
  --sam-file-type arg                  File type for SAM metadata.
532 16 Gianluca Petrillo
  --sam-data-tier arg                  SAM data tier spec-label>:<tier-spec>).
533 16 Gianluca Petrillo
  --sam-run-type arg                   Global run-type for SAM metadata.
534 16 Gianluca Petrillo
  --sam-stream-name arg                SAM stream name (<module-label>:<stream-
535 16 Gianluca Petrillo
                                       name>).
536 7 Gianluca Petrillo
537 7 Gianluca Petrillo
Art has completed and will exit with status 1.
538 1 Erica Snider
</pre>
539 7 Gianluca Petrillo
540 7 Gianluca Petrillo
h2. Running a Job
541 7 Gianluca Petrillo
542 7 Gianluca Petrillo
To run the job defined by the script above, do
543 1 Erica Snider
<pre>
544 1 Erica Snider
$ lar -c job/prodgenie.fcl
545 7 Gianluca Petrillo
</pre>
546 7 Gianluca Petrillo
547 7 Gianluca Petrillo
One can stop a job in two ways:
548 7 Gianluca Petrillo
# press <Ctrl>+<C> once: the job will complete at the end of the current module. If the job is running in the background type @kill -9 %jobID@ on the command line.
549 1 Erica Snider
# press <Ctrl>+<C> twice: the job should stop immediately and, depending on the shall settings, might produce a core dump.
550 7 Gianluca Petrillo
551 7 Gianluca Petrillo
If you want to have your job keep running even if you get disconnected from a remote session, depending on your shell you might need to start it with:
552 7 Gianluca Petrillo
<pre>
553 7 Gianluca Petrillo
$ nohup lar job/prodgenie.fcl >& pg.out
554 1 Erica Snider
</pre>
555 7 Gianluca Petrillo
556 7 Gianluca Petrillo
To stop such a job, then do
557 7 Gianluca Petrillo
<pre>
558 7 Gianluca Petrillo
$ pgrep lar # to find the job ID (use ps to disambiguate if more than one lar process are running)
559 7 Gianluca Petrillo
$ kill -INT jobID
560 7 Gianluca Petrillo
</pre>
561 1 Erica Snider
562 1 Erica Snider
One can print out the configuration of the job without starting the executable by:
563 1 Erica Snider
<pre>
564 1 Erica Snider
$ ART_DEBUG_CONFIG=1 lar -c prodgenie.fcl
565 1 Erica Snider
</pre>
566 1 Erica Snider
in @bash@, or
567 1 Erica Snider
<pre>
568 1 Erica Snider
> env ART_DEBUG_CONFIG=1 lar -c prodgenie.fcl
569 1 Erica Snider
</pre>
570 1 Erica Snider
in C-shell, which produces the output
571 1 Erica Snider
<pre>
572 1 Erica Snider
** ART_DEBUG_CONFIG is defined: config debug output follows **
573 1 Erica Snider
all_modules: [ "out1"
574 1 Erica Snider
             , "daq"
575 1 Erica Snider
             , "generator"
576 1 Erica Snider
             , "largeant"
577 1 Erica Snider
             , "rns"
578 1 Erica Snider
             ]
579 1 Erica Snider
outputs: { out1: { fileName: "genie_gen.root"
580 1 Erica Snider
                   module_label: "out1"
581 1 Erica Snider
                   module_type: "RootOutput"
582 1 Erica Snider
                 }
583 1 Erica Snider
         }
584 1 Erica Snider
physics: { end_paths: [ "stream1" ]
585 1 Erica Snider
           producers: { daq: { Col3DCorrection: 2.5
586 1 Erica Snider
                               ColFieldRespAmp: 3.54e-2
587 1 Erica Snider
                               CompressionType: "none"
588 1 Erica Snider
                               DriftEModuleLabel: "largeant"
589 1 Erica Snider
                               FieldBins: 75
590 7 Gianluca Petrillo
                               Ind3DCorrection: 1.5
591 7 Gianluca Petrillo
                               IndFieldRespAmp: 1.8e-2
592 1 Erica Snider
                               LowCutoff: 7.5
593 1 Erica Snider
                               NoiseFact: 1.32e-1
594 7 Gianluca Petrillo
                               NoiseWidth: 6.24e1
595 1 Erica Snider
                               ResponseFile: "shape-argo.root"
596 1 Erica Snider
                               ShapeTimeConst: [ 3000
597 1 Erica Snider
                                               , 900
598 1 Erica Snider
                                               ]
599 7 Gianluca Petrillo
                               module_label: "daq"
600 1 Erica Snider
                               module_type: "SimWireT962"
601 1 Erica Snider
                             }
602 8 Gianluca Petrillo
                        generator: { BeamCenter: [ -1400
603 1 Erica Snider
                                                 , -350
604 8 Gianluca Petrillo
                                                 , 0
605 1 Erica Snider
                                                 ]
606 8 Gianluca Petrillo
[...]
607 1 Erica Snider
</pre>
608 8 Gianluca Petrillo
609 8 Gianluca Petrillo
This functionality is particularly helpful when trying to debug what input parameters were passed to the job.
610 8 Gianluca Petrillo
611 8 Gianluca Petrillo
h2. Why did my job fail?
612 8 Gianluca Petrillo
613 8 Gianluca Petrillo
In case of failure, @art@ usually provides some information about the failure.
614 8 Gianluca Petrillo
615 8 Gianluca Petrillo
If a job fails, look at the warnings printed to the screen or any output log files.
616 8 Gianluca Petrillo
617 1 Erica Snider
Also check in the [[Breaking_Changes|breaking changes]] page for information about problems (and solutions) caused by changes to the code.
618 1 Erica Snider
619 17 Gianluca Petrillo
You can ask help to the larsoft@fnal.gov mailing list, or open a issue in the appropriate tracker (usually "the LArSoft one":https://cdcvs.fnal.gov/redmine/projects/larsoft/issues), or to art-users@fnal.gov .
620 1 Erica Snider
The usual bug report rules apply: include at least
621 1 Erica Snider
* the full command line of the failing command
622 1 Erica Snider
* a full path to the input files
623 1 Erica Snider
* which LArSoft release you are using
624 1 Erica Snider
* the complete error message.
625 1 Erica Snider
626 17 Gianluca Petrillo
If a bug should be reported to the art-users@fnal.gov list, attach the _complete output of the job_ to the email.
627 1 Erica Snider
628 1 Erica Snider
h2. Submitting Jobs to the compute farms
629 1 Erica Snider
630 1 Erica Snider
We have to find out where the page with the submission instructions went (*TODO*!).