Project

General

Profile

Running Jobs » History » Version 20

Gianluca Petrillo, 01/26/2016 12:00 PM

1 1 Erica Snider
{{toc}}
2 1 Erica Snider
3 1 Erica Snider
4 1 Erica Snider
h1. Running Jobs
5 1 Erica Snider
6 9 Gianluca Petrillo
This page describes the job configuration file (often, in jargon: _FHiCL file_) and how to run a job using one.
7 1 Erica Snider
8 3 Gianluca Petrillo
It is assumed that you have previously read the [[ Quick-start guide to using and developing LArSoft code ]] and the information on [[Using LArSoft on the GPVM nodes]].
9 1 Erica Snider
10 11 Gianluca Petrillo
h2. The Job Configuration
11 1 Erica Snider
12 9 Gianluca Petrillo
Once a base release is set up, it is easy to run a job. The basic unit for running a job is the job-control script, written in the FHICL language. The FHICL language provides a simple mechanism for including parameter set configurations from different files such that many job configuration files can use the same configuration for a module or service.  
13 1 Erica Snider
14 10 Gianluca Petrillo
There is also a nice "FHiCL quick start guide":https://cdcvs.fnal.gov/redmine/attachments/16021/quick_start.pdf available for more details. 
15 2 Lynn Garren
16 3 Gianluca Petrillo
17 10 Gianluca Petrillo
h3. Key Concepts in FHiCL
18 1 Erica Snider
19 1 Erica Snider
There are a few key concepts to writing a FHICL job control script.  In order, they are
20 1 Erica Snider
21 1 Erica Snider
# Including previously defined configurations for services and modules from other files.  This is done using @#include@ statements. *Be sure you don't have any trailing space or tab characters* on the @#include@ line.
22 9 Gianluca Petrillo
# Service block, denoted by @services: { }@  This block will contain configurations for ART specific services such as the @TFileService@ and the @RandomNumberGenerator@.  It also contains the configuration of LArSoft specific services[1].
23 9 Gianluca Petrillo
# Source block, denoted by @source: { }@.  This block tells the job what kind of source to expect (@EmptyEvent@ in the case of Monte Carlo generation, @RootInput@ in the case of anything downstream of a Monte Carlo generator or reconstruction, and specific modules for data from the detector), the file name for the input source if appropriate, and how many events to process.  Both the file name and number of events to process can be specified on the command line.
24 9 Gianluca Petrillo
# Output block, denoted by @outputs: { }@  This block tells the job what kind of output to make, i.e. @RootOutput@, and what the name of the output file should be.  The output file name can be specified on the command line.  It is possible to define more than one output file if one wanted to run a job that produced different output files based on filter criteria - i.e. empty events are put in one file and events with neutrinos in them are put in another. Multiple output files can only be specified in the job configuration file, not from the command line.
25 9 Gianluca Petrillo
# Physics block, denoted by @physics: { }@  This block is where all producer, analyzer, and filter modules are configured.  Sequences of producer and filter modules to run is defined in user-named _paths_ in this block.  The list of analyzers and output modules to run is defined in a separate user-named path.  The block also defines two keyword parameters, @trigger_paths@ and @end_paths@.  @trigger_paths@ contains all producer and filter paths to run, and @end_paths@ contains the analyzer and output path.
26 3 Gianluca Petrillo
27 10 Gianluca Petrillo
Comments may be included in FHiCL configuration files using the "#" character.  The @#include@ is a keyword so that the parser knows not to ignore what comes after "#include[2]".
28 9 Gianluca Petrillo
29 9 Gianluca Petrillo
fn1. In old FHiCL files you will notice that LArSoft and in general non-art service configuration is enclosed in a @user@ block, that is now deprecated.
30 9 Gianluca Petrillo
31 9 Gianluca Petrillo
fn2. Note that the FHiCL parser can't recognize a comment on the same line as a @#include@ directive.
32 1 Erica Snider
33 3 Gianluca Petrillo
34 10 Gianluca Petrillo
h3. FHiCL Rules
35 1 Erica Snider
36 10 Gianluca Petrillo
There are a few of rules to keep in mind about FHiCL:
37 10 Gianluca Petrillo
* The value of the @process_name@ parameter may not contain underscores as the process name is used in the ROOT file branch name.  Module labels may not contain underscores either, for the same reason.
38 1 Erica Snider
* Parameter set names may not contain numbers, periods, backslashes, stars, etc. They may contain underscores.
39 3 Gianluca Petrillo
* Put the values for all string parameters in double quotes, @"..."@
40 3 Gianluca Petrillo
* Specify input vectors using @[ , , ]@, i.e. if you want a vector of doubles do @MyVector: [1.0, 3e-9, -900.]@
41 10 Gianluca Petrillo
* You pick out configurations from the @PROLOG@ section(s), usually defined in the @#include@ files, using the @ @local::@ syntax.  The value after the "@::@" is the name of the configuration specified in the @PROLOG@ (see the next bullet)
42 3 Gianluca Petrillo
* You can override the value of an included configuration. For example, imagine there is a configuration specified in a included file called @mymoduleconfig@ and it contains the value @-5@ for the parameter named @myint@. One can load the configuration and then change the value of @myint@ by doing the following:
43 3 Gianluca Petrillo
*# inside the producers block:
44 1 Erica Snider
<pre>
45 3 Gianluca Petrillo
physics: {
46 3 Gianluca Petrillo
  producers: {
47 3 Gianluca Petrillo
    # ...
48 3 Gianluca Petrillo
    mymod: @local::mymoduleconfig
49 3 Gianluca Petrillo
  }
50 3 Gianluca Petrillo
}
51 3 Gianluca Petrillo
</pre>
52 3 Gianluca Petrillo
*# out of the physics block 
53 3 Gianluca Petrillo
<pre>
54 1 Erica Snider
physics.producer.mymod.myint: 1
55 1 Erica Snider
</pre>
56 3 Gianluca Petrillo
The last value for a parameter always wins. If the second line was repeated with the value @2@ instead of @1@, the job would run with @myint@ as @2@.
57 3 Gianluca Petrillo
Also note that in the example the original content of @mymoduleconfig@ is not changed when the content of @mymod@ is.
58 3 Gianluca Petrillo
59 6 Gianluca Petrillo
h3. Example configuration file: @detsimmodules.fcl@
60 6 Gianluca Petrillo
61 6 Gianluca Petrillo
An example of a file with predefined configurations for modules is in the "larsim/DetSim/detsimmodules.fcl":https://cdcvs.fnal.gov/redmine/projects/larsim/repository/revisions/develop/entry/DetSim/detsimmodules.fcl file.
62 1 Erica Snider
All the definitions are inside a prologue block.
63 18 Gianluca Petrillo
The following is an excerpt taken from LArSoft @v05_00_00@ (experiments now use their specific @detsimmodule_Xxxx.fcl@ configuration file though):
64 6 Gianluca Petrillo
65 1 Erica Snider
<pre>
66 1 Erica Snider
BEGIN_PROLOG
67 1 Erica Snider
68 18 Gianluca Petrillo
standard_simwire:
69 18 Gianluca Petrillo
{
70 18 Gianluca Petrillo
  module_type:        "SimWire"
71 18 Gianluca Petrillo
  DriftEModuleLabel:  "largeant"
72 18 Gianluca Petrillo
  NoiseFact:           0.0132      # Noise Scale
73 18 Gianluca Petrillo
  NoiseWidth:         62.4         # Exponential Noise width (kHz)
74 18 Gianluca Petrillo
  LowCutoff:           7.5         # Low frequency filter cutoff (kHz)
75 18 Gianluca Petrillo
  FieldBins:          75
76 18 Gianluca Petrillo
  Col3DCorrection:    2.5
77 18 Gianluca Petrillo
  Ind3DCorrection:    1.5
78 18 Gianluca Petrillo
  ColFieldRespAmp:    0.0354
79 18 Gianluca Petrillo
  IndFieldRespAmp:    0.018
80 18 Gianluca Petrillo
  ShapeTimeConst:     [ 3000., 900. ]
81 18 Gianluca Petrillo
  CompressionType:    "none"
82 6 Gianluca Petrillo
}
83 6 Gianluca Petrillo
84 6 Gianluca Petrillo
argoneut_simwireana: @local::standard_simwireana
85 6 Gianluca Petrillo
86 18 Gianluca Petrillo
###
87 18 Gianluca Petrillo
### ... and more configurations ...
88 18 Gianluca Petrillo
###
89 6 Gianluca Petrillo
90 6 Gianluca Petrillo
END_PROLOG
91 6 Gianluca Petrillo
</pre>
92 18 Gianluca Petrillo
93 6 Gianluca Petrillo
94 5 Gianluca Petrillo
h3. Configuring the "message service":https://cdcvs.fnal.gov/redmine/projects/messagefacility/wiki/Using_MessageFacility#Using-MessageFacility
95 1 Erica Snider
96 4 Gianluca Petrillo
Several standard configurations for the message service are in "lardata/Utilities/messageservice.fcl":https://cdcvs.fnal.gov/redmine/projects/lardata/repository/revisions/develop/entry/Utilities/messageservice.fcl.  There is one configuration for each level of message output - Debug, Info, Warning, and Error.  These configurations will be applied to each message level that is specified and those of higher priority.  For example, the Info configuration will print out Info, Warning and Error level messages while the Warning configuration only print outs Warning and Error level messages. The "standard" debug configurations will cause the messages to go to a specified output file, @debug.log@. The Error configuration redirect to standard error stream (like @std::cerr@), while the others print into the standard output (@std::cout@). All impose some limits on the repetition of some frequent messages.
97 5 Gianluca Petrillo
Remember that to use one of these "standard" configurations you need to include it in your FCL file: they are standard, not default.
98 4 Gianluca Petrillo
If you want to define your own configuration, please take a look at the comments in "lardata/Utilities/messageservice.fcl":https://cdcvs.fnal.gov/redmine/projects/lardata/repository/revisions/develop/entry/Utilities/messageservice.fcl file to determine how to do so.  
99 1 Erica Snider
100 1 Erica Snider
Examples of how to include the usual use of the message service configurations are in the example files below.
101 1 Erica Snider
102 5 Gianluca Petrillo
To get a different level of output from just one module (say @DBSCAN@) one would do:
103 1 Erica Snider
104 1 Erica Snider
<pre>
105 1 Erica Snider
services:
106 1 Erica Snider
{
107 1 Erica Snider
  # Load the service that manages root files for histograms.
108 1 Erica Snider
  TFileService: { fileName: "reco_hist.root" }
109 1 Erica Snider
  Timing:       {}
110 1 Erica Snider
  RandomNumberGenerator: {} #ART native random number generator
111 1 Erica Snider
112 1 Erica Snider
  # configure the message service with the INFO for DBSCAN
113 1 Erica Snider
  # and WARNING level for everything else
114 1 Erica Snider
  message: { 
115 1 Erica Snider
    destinations: {  
116 1 Erica Snider
      infomsg: {
117 1 Erica Snider
        type: "cout"
118 1 Erica Snider
        threshold: "INFO"
119 1 Erica Snider
        append: true
120 1 Erica Snider
        category: {
121 1 Erica Snider
           DBSCAN: {
122 1 Erica Snider
             reportEvery: 1
123 1 Erica Snider
           }
124 1 Erica Snider
        }
125 1 Erica Snider
      }
126 1 Erica Snider
      warningmsg: {
127 1 Erica Snider
        type:      "cout"	
128 1 Erica Snider
        threshold: "WARNING"
129 1 Erica Snider
        append:    true        
130 1 Erica Snider
        categories: {
131 1 Erica Snider
          default: {
132 1 Erica Snider
            limit:       1000   
133 1 Erica Snider
            timespan:    60    
134 1 Erica Snider
          }
135 1 Erica Snider
        } # end categories
136 1 Erica Snider
      } # end warningmsg
137 1 Erica Snider
    } # end destinations
138 1 Erica Snider
  } # end standard_warning
139 1 Erica Snider
140 1 Erica Snider
  user:         @local::argoneut_services 	     
141 1 Erica Snider
}
142 1 Erica Snider
</pre>
143 1 Erica Snider
144 12 Gianluca Petrillo
h4. Debug messages
145 12 Gianluca Petrillo
146 12 Gianluca Petrillo
Note that debugging messages are treated in a special way:
147 12 Gianluca Petrillo
# there are two ways to print a debug message in the code: by using @mf::LogDebug@, and by using @LOG_DEBUG@ macro; the messages using the latter will not be present when a non-debug qualifier (@prof@ and @opt@) is used. In fact, no code is generated at all from a @LOG_DEBUG@ call unless using a @debug@ qualified build
148 12 Gianluca Petrillo
# debug output from modules is selectively enabled by the @debugModules@ list; to enable all the messages, use @debugModules: [ "*" ]@
149 12 Gianluca Petrillo
150 12 Gianluca Petrillo
Also remember that, depending on the configuration, the debug output could be not shown on screen, but only stored in a file (e.g. @debug.log@).
151 1 Erica Snider
152 1 Erica Snider
h3. Example job script: @prodgenie.fcl@
153 5 Gianluca Petrillo
154 19 Katherine Lato
To see an example job script to produce Monte Carlo events, go to: "larsim/EventGenerator/GENIE/prodgenie.fcl":https://cdcvs.fnal.gov/redmine/projects/larsim/repository/revisions/develop/entry/EventGenerator/GENIE/prodgenie.fcl .  The job defined by this script will generate neutrino interactions using GENIE, run them through Geant4, do the electron transport and then simulate the electronics.
155 1 Erica Snider
156 1 Erica Snider
157 1 Erica Snider
h3. How to override a default parameter
158 1 Erica Snider
159 1 Erica Snider
If you want to override a default parameter that has been included from a predefined parameter set, you must specify which parameter and its value as
160 1 Erica Snider
161 7 Gianluca Petrillo
<pre>
162 1 Erica Snider
mainBlock.subBlock.label.parameterName: newValue
163 7 Gianluca Petrillo
</pre>
164 1 Erica Snider
165 1 Erica Snider
where 
166 1 Erica Snider
167 13 Gianluca Petrillo
* @mainBlock@ can be @services@ or @physics@ 
168 13 Gianluca Petrillo
* @subBlock@ can be @producers@, @filters@, or @analyzers@, skipped in case of services, and @user@ for old service configuration
169 13 Gianluca Petrillo
* @label@ is the name of the desired service or module instance in a producers, filters, or analyzers block
170 1 Erica Snider
* @parameterName@ is the name of the desired parameter
171 13 Gianluca Petrillo
* @newValue@ is the desired new value; a list or entire block (that is, a brace-enclosed sequence of @key: value@ pairs) can be specified
172 1 Erica Snider
173 1 Erica Snider
These lines must go after the @mainBlock@ and be outside of any other block.
174 1 Erica Snider
175 13 Gianluca Petrillo
For example, if one wanted to change the default value of the @HitsModuleLabel@ parameter in the @dbcluster@ module instance in the previous section, one would put
176 1 Erica Snider
<pre>
177 13 Gianluca Petrillo
physics.producers.dbcluster.HitsModuleLabel: "differentHitModuleLabel"
178 1 Erica Snider
</pre>
179 13 Gianluca Petrillo
Note that FHiCL allows for completely replacing a list but not for changing its size (i.e., it's not possible to add an element to an existing list).
180 1 Erica Snider
181 13 Gianluca Petrillo
This behavior is general: FHiCL only allows to replace elements. For example, if one wants to change both @eps@ and @epstwo@ parameters,
182 13 Gianluca Petrillo
<pre>
183 13 Gianluca Petrillo
physics.producers.dbcluster.DBScanAlg: { # WRONG!!!
184 13 Gianluca Petrillo
  eps:    1.1
185 13 Gianluca Petrillo
  epstwo: 1.6
186 13 Gianluca Petrillo
}
187 14 Gianluca Petrillo
</pre> will not work, because it is replacing the whole @physics.producers.dbcluster.DBScanAlg@ parameter set, thus removing e.g. the parameter @minPts@.
188 13 Gianluca Petrillo
The correct way to change parameters is to think of that as _replacing_ them, and to point exactly at each of them:
189 13 Gianluca Petrillo
<pre>
190 1 Erica Snider
physics.producers.dbcluster.DBScanAlg.eps:    1.1
191 13 Gianluca Petrillo
physics.producers.dbcluster.DBScanAlg.epstwo: 1.6
192 14 Gianluca Petrillo
</pre> will leave the other parameters (e.g., @minPts@) unchanged.
193 7 Gianluca Petrillo
194 1 Erica Snider
h3. fhicl Emacs syntax highlighting
195 1 Erica Snider
196 7 Gianluca Petrillo
If you use Emacs as your editor, you can put the following into your @.emacs@ file in your home directory to cause it to display @.fcl@ files with syntax highlighting
197 1 Erica Snider
198 1 Erica Snider
<pre>
199 1 Erica Snider
(setq fclKeywords
200 1 Erica Snider
'(
201 1 Erica Snider
  ;; This, due to poor language design, conflicts with comments and fails
202 1 Erica Snider
  ("#include" . font-lock-keyword-face)
203 1 Erica Snider
  ("@local" . font-lock-keyword-face)
204 1 Erica Snider
  ;; All these names are magic, I think
205 1 Erica Snider
206 1 Erica Snider
207 1 Erica Snider
("process_name:\\|services:\\|source:\\|outputs:\\|physics\\|producers:\\|filters:\\|analyzers:" . font-lock-builtin-face)
208 1 Erica Snider
  ("true\\|false" . font-lock-builtin-face)
209 1 Erica Snider
  ;; Variable definitions are followed by colons
210 1 Erica Snider
211 1 Erica Snider
  ("[a-zA-Z0-9_]*:" . font-lock-variable-name-face)
212 1 Erica Snider
 )
213 1 Erica Snider
)
214 1 Erica Snider
215 1 Erica Snider
;; Python mode gets us comment handling and indentation at colons
216 1 Erica Snider
217 1 Erica Snider
(define-derived-mode fcl-mode python-mode
218 1 Erica Snider
 (setq mode-name "FHICL")
219 1 Erica Snider
 (setq font-lock-defaults '(fclKeywords))
220 1 Erica Snider
;;  (setq tab-width 2) ;; Doesn't seem to work
221 1 Erica Snider
222 1 Erica Snider
)
223 1 Erica Snider
224 1 Erica Snider
(add-to-list 'auto-mode-alist '("\\.fcl\\'" . fcl-mode))
225 1 Erica Snider
</pre>
226 1 Erica Snider
227 15 Gianluca Petrillo
bq. Please use _only spaces_ for alignment: it's not the perfect solution, but it makes people using different editors see the same code alignment.
228 7 Gianluca Petrillo
229 1 Erica Snider
h2. Executable and command line options
230 1 Erica Snider
231 7 Gianluca Petrillo
Currently there is one executable to run in LArSoft. The executable to run a typical reconstruction or analysis job is @lar@ which is placed in the user's path by the setup script. To see what options are available do
232 7 Gianluca Petrillo
<pre>
233 7 Gianluca Petrillo
$ lar -h
234 7 Gianluca Petrillo
</pre>
235 20 Gianluca Petrillo
Some of the most useful options (from @lar@ version @v1_17_05@) are:
236 20 Gianluca Petrillo
* @-c@/@--config@ @configfile@: specify the job configuration (FHiCL) file
237 20 Gianluca Petrillo
* @--print-available@ what: list all available plugins: @module@, @service@, @source@, @plugin@
238 20 Gianluca Petrillo
* @--print-description@ what: print description of specified @module@, @service@, @source@, or other plugin
239 20 Gianluca Petrillo
* @-s@/@--source@ file: source ROOT data file (may be specified more than once)
240 20 Gianluca Petrillo
* @-S@/@--source-list@ filelist: text file containing a list of source files to read, one per line
241 20 Gianluca Petrillo
* @-n@/@--nevts@ number: number of events to process
242 20 Gianluca Petrillo
* @--nskip@ number: number of events to skip
243 20 Gianluca Petrillo
* @--rethrow-all@: all exceptions are propagated rather than handled by art
244 20 Gianluca Petrillo
* @--config-out@ newfile: before running the job, creates a text file with the complete job configuration
245 1 Erica Snider
246 7 Gianluca Petrillo
247 7 Gianluca Petrillo
h2. Running a Job
248 7 Gianluca Petrillo
249 7 Gianluca Petrillo
To run the job defined by the script above, do
250 1 Erica Snider
<pre>
251 1 Erica Snider
$ lar -c job/prodgenie.fcl
252 7 Gianluca Petrillo
</pre>
253 7 Gianluca Petrillo
254 7 Gianluca Petrillo
One can stop a job in two ways:
255 7 Gianluca Petrillo
# press <Ctrl>+<C> once: the job will complete at the end of the current module. If the job is running in the background type @kill -9 %jobID@ on the command line.
256 1 Erica Snider
# press <Ctrl>+<C> twice: the job should stop immediately and, depending on the shall settings, might produce a core dump.
257 7 Gianluca Petrillo
258 7 Gianluca Petrillo
If you want to have your job keep running even if you get disconnected from a remote session, depending on your shell you might need to start it with:
259 7 Gianluca Petrillo
<pre>
260 7 Gianluca Petrillo
$ nohup lar job/prodgenie.fcl >& pg.out
261 1 Erica Snider
</pre>
262 7 Gianluca Petrillo
263 7 Gianluca Petrillo
To stop such a job, then do
264 7 Gianluca Petrillo
<pre>
265 7 Gianluca Petrillo
$ pgrep lar # to find the job ID (use ps to disambiguate if more than one lar process are running)
266 7 Gianluca Petrillo
$ kill -INT jobID
267 7 Gianluca Petrillo
</pre>
268 1 Erica Snider
269 1 Erica Snider
One can print out the configuration of the job without starting the executable by:
270 1 Erica Snider
<pre>
271 1 Erica Snider
$ ART_DEBUG_CONFIG=1 lar -c prodgenie.fcl
272 1 Erica Snider
</pre>
273 1 Erica Snider
in @bash@, or
274 1 Erica Snider
<pre>
275 1 Erica Snider
> env ART_DEBUG_CONFIG=1 lar -c prodgenie.fcl
276 1 Erica Snider
</pre>
277 1 Erica Snider
in C-shell, which produces the output
278 1 Erica Snider
<pre>
279 1 Erica Snider
** ART_DEBUG_CONFIG is defined: config debug output follows **
280 1 Erica Snider
all_modules: [ "out1"
281 1 Erica Snider
             , "daq"
282 1 Erica Snider
             , "generator"
283 1 Erica Snider
             , "largeant"
284 1 Erica Snider
             , "rns"
285 1 Erica Snider
             ]
286 1 Erica Snider
outputs: { out1: { fileName: "genie_gen.root"
287 1 Erica Snider
                   module_label: "out1"
288 1 Erica Snider
                   module_type: "RootOutput"
289 1 Erica Snider
                 }
290 1 Erica Snider
         }
291 1 Erica Snider
physics: { end_paths: [ "stream1" ]
292 1 Erica Snider
           producers: { daq: { Col3DCorrection: 2.5
293 1 Erica Snider
                               ColFieldRespAmp: 3.54e-2
294 1 Erica Snider
                               CompressionType: "none"
295 1 Erica Snider
                               DriftEModuleLabel: "largeant"
296 1 Erica Snider
                               FieldBins: 75
297 7 Gianluca Petrillo
                               Ind3DCorrection: 1.5
298 7 Gianluca Petrillo
                               IndFieldRespAmp: 1.8e-2
299 1 Erica Snider
                               LowCutoff: 7.5
300 1 Erica Snider
                               NoiseFact: 1.32e-1
301 7 Gianluca Petrillo
                               NoiseWidth: 6.24e1
302 1 Erica Snider
                               ResponseFile: "shape-argo.root"
303 1 Erica Snider
                               ShapeTimeConst: [ 3000
304 1 Erica Snider
                                               , 900
305 1 Erica Snider
                                               ]
306 7 Gianluca Petrillo
                               module_label: "daq"
307 1 Erica Snider
                               module_type: "SimWireT962"
308 1 Erica Snider
                             }
309 8 Gianluca Petrillo
                        generator: { BeamCenter: [ -1400
310 1 Erica Snider
                                                 , -350
311 8 Gianluca Petrillo
                                                 , 0
312 1 Erica Snider
                                                 ]
313 8 Gianluca Petrillo
[...]
314 1 Erica Snider
</pre>
315 8 Gianluca Petrillo
316 8 Gianluca Petrillo
This functionality is particularly helpful when trying to debug what input parameters were passed to the job.
317 8 Gianluca Petrillo
318 8 Gianluca Petrillo
h2. Why did my job fail?
319 8 Gianluca Petrillo
320 8 Gianluca Petrillo
In case of failure, @art@ usually provides some information about the failure.
321 8 Gianluca Petrillo
322 8 Gianluca Petrillo
If a job fails, look at the warnings printed to the screen or any output log files.
323 8 Gianluca Petrillo
324 1 Erica Snider
Also check in the [[Breaking_Changes|breaking changes]] page for information about problems (and solutions) caused by changes to the code.
325 1 Erica Snider
326 17 Gianluca Petrillo
You can ask help to the larsoft@fnal.gov mailing list, or open a issue in the appropriate tracker (usually "the LArSoft one":https://cdcvs.fnal.gov/redmine/projects/larsoft/issues), or to art-users@fnal.gov .
327 1 Erica Snider
The usual bug report rules apply: include at least
328 1 Erica Snider
* the full command line of the failing command
329 1 Erica Snider
* a full path to the input files
330 1 Erica Snider
* which LArSoft release you are using
331 1 Erica Snider
* the complete error message.
332 1 Erica Snider
333 17 Gianluca Petrillo
If a bug should be reported to the art-users@fnal.gov list, attach the _complete output of the job_ to the email.
334 1 Erica Snider
335 1 Erica Snider
h2. Submitting Jobs to the compute farms
336 1 Erica Snider
337 1 Erica Snider
We have to find out where the page with the submission instructions went (*TODO*!).