Creating cross-section spline files

Generally users should generate events using the splines that exist in the nusoft UPS area; these in conjunction with the GENIE installation in the same area ensures a consistent, reproducible setup. Personal installations are strongly discouraged and certainly should not be used for any general production. If a special setup is needed then a steps should be taken to get it installed in the nusoft UPS area for reference. If the spline files are missing needed isotopes please contact Robert Hatcher so they can be generated.

This page describes a set of scripts written to wrapper the details of how gxspl files are created. These scripts are tuned for use at FNAL (file paths) and with condor (batch job processing). They are meant for general use by the FNAL neutrino community; they are intended to be user and experiment agnostic. Users should not copy the scripts to alternative locations, but use them as they are -- this will avoid version skew and confusion about modified versions. If there are problems running them (due to path issues or such) then please contact Robert Hatcher to have them modified to be compatible with your setup. Copies in alternative locations will not be supported.

Currently the process is broken into six steps:

  • initialize the script and output directories
  • stage 1: generate individual neutrino on free nucleon splines
  • stage 2: combine all free nucleon splines
  • stage 3: generate spines for all neutrinos on a specific isotope
  • stage 4: combine all isotope files
  • stage 5: reduce file to only the (current) minimum needed isotopes

While the 5 stages of processing could be joined together with a DAG (Directed Acyclical Graph) they currently need manual user intervention to get started. Jobs for later stages should not be submitted until the prior steps are complete.

The end result should be a file named gxspl-NuMIsmall.xml. Care must be taken to ensure that information about how this file was created is not lost. A gxspl file is only valid for use by GENIE for the exact same release and configuration of GENIE as was used to create it. If you use a cross-section spline file in a different release or one with a different configuration your results may be inconsistent. GENIE has no way of validating whether there is an inconsistent usage, so it is important to maintain awareness. It is recommended that users rename the generic gxspl-NuMI*.xml files to embed relevant information into the filename to at least indicate a non-standard file.

Preparing a setup:

The script to initialize the process is located at: /grid/fermiapp/nusoft/generate_genie_xsec/

Setting a user script and output location

These scripts are written to allow them to be run on the (FNAL) "grid" under the condor batch system. This imposes restrictions on where scripts can be run from and files written to. The user must supply a subpath using the -o flag to designate the directory structure.

$ /grid/fermiapp/nusoft/generate_genie_xsec/ -o subpath
  • /expt/app/users/user/subpath/ is used for scripts
  • /expt/data/users/user/subpath/ is used for output XML files and logs (for NOvA only: /nova/ana/users/...)

The subpath needn't be a single name but can be something like: my_genie_xsec/modified_MA/MA1.0. The expt and user are inferred from the id of the user running the script. The id group to BlueArc mounting point has been verified for MINOS and NOvA but may not be correct for other experiments; please let me know what it needs to be if it doesn't work as-is.

Telling the scripts how to get the GENIE executables

These scripts need the GENIE executables gmkspl and gspladd to do the work. Thus a working version of GENIE needs to be available on the grid, with all the ancillary packages (e.g. ROOT, log4cpp, LHAPDF, etc.) configured. For MINOS and NOvA (currently) this is built into the scripts by setting up their respective offline environments using -s minos or -s nova. Specification of a release can be done using the -S flag, eg. -S "-r development"; quotes are required for cases where there are multiple arguments. For other experiments or non-standard installations one can use the -s flag to identify a bash script that, when sourced, will setup the desired GENIE release. If such a script is written to accept arguments to differentiate between releases it too can use the -S flag to pass those arguments to it.

Examples of

$ /grid/fermiapp/nusoft/generate_genie_xsec/ -o my_genie_xsec/modified -s /path/to/ -S "-r myversion" 

generates the screen output:

### setup and conditions script for my_genie_xsec/modified
# how to setup GENIE
SETUPCMD="source /path/to/ -r myversion" 
# set the environment
export APPPATH=/grid/fermiapp/nusoft/generate_genie_xsec
export USERAPPPATH=/nova/app/users/rhatcher/my_genie_xsec/modified
export USERDATAPATH=/nova/ana/users/rhatcher/my_genie_xsec/modified
export STAGE1RESULTS=/nova/ana/users/rhatcher/my_genie_xsec/modified/work/freenuc
export STAGE3RESULTS=/nova/ana/users/rhatcher/my_genie_xsec/modified/work/element
export GXMLPATH=.:/nova/app/users/rhatcher/my_genie_xsec/modified:/nova/data/users/rhatcher/my_genie_xsec/modified
export GMSGLAYOUT=SIMPLE #[BASIC] or timestamp-less SIMPLE
export JOBSUB=jobsub
export NKNOTS=500
export EMIN=0.01
export EMAX=200.00
# end-of-setup script
to run: 
   $ cd /nova/app/users/rhatcher/my_genie_xsec/modified
follow the instructions

The first part of the output displays the selected configuration. The to run: then directs the user to the configured location and the script to run.

On systems where setting up genie is part of the offline framework, one can simply supply the experiment name (i.e. nova or minos) for the -s flag.

Using NOvA is odd in two ways. First the setup requires more than sourcing a single script, so we make a wrapper script in the new location and source that. Secondly, their user-writable output disk is not /nova/data but /nova/ana

$ /grid/fermiapp/nusoft/generate_genie_xsec/ -o my_genie_xsec/modified -s nova -S "-r S12.05.31" 

SETUPCMD="source /nova/app/users/rhatcher/my_genie_xsec/modified/ " 

Running the stages

Initial state after

$ ls /nova/app/users/rhatcher/my_genie_xsec/modified
$ ls /nova/data/users/rhatcher/my_genie_xsec/modified

If you are planning on running GENIE with a modified set of parameters then at this point copy the UserPhysicsOptions.xml to the corresponding data directory The GXMLPATH variable will be set to include the data directory, so this will get picked up automatically if the file is placed there.

Running the in the app directory without args will check the current state of processing and suggest what step to take to proceed:

$ cd /nova/app/users/rhatcher/my_genie_xsec/modified
$ ./ 
checking status

checking stage 1 in /nova/data/users/rhatcher/my_genie_xsec/modified/work/freenuc:
  missing: [-P  0] gxspl-nu_tau_bar-freep.xml
  missing: [-P  1] gxspl-nu_tau_bar-freen.xml
  missing: [-P  2] gxspl-nu_mu_bar-freep.xml
  missing: [-P  3] gxspl-nu_mu_bar-freen.xml
  missing: [-P  4] gxspl-nu_e_bar-freep.xml
  missing: [-P  5] gxspl-nu_e_bar-freen.xml
  missing: [-P  6] gxspl-nu_e-freep.xml
  missing: [-P  7] gxspl-nu_e-freen.xml
  missing: [-P  8] gxspl-nu_mu-freep.xml
  missing: [-P  9] gxspl-nu_mu-freen.xml
  missing: [-P 10] gxspl-nu_tau-freep.xml
  missing: [-P 11] gxspl-nu_tau-freen.xml
stage 1:  have 0, missing 12
to start processing this stage:
  cd /nova/app/users/rhatcher/my_genie_xsec/modified
  jobsub -g -N 12 -S 1

At any time, running ./ will check the current status of processing. Only proceed with the jobsub after checking that all the jobs from the previous submission are complete. If in a multi-job stage some of the jobs failed to complete (or are still running) then the will indicate how to run individual make-up jobs (substitute the # following the -P for #). This check
does not know how to check whether condor jobs are still running so it might suggest running make-up jobs when a stage is incomplete; it is thus important that the user know the status of their jobs.

When all steps are complete it will look like:

$ ./ 
checking status

checking stage 1 in /minos/data/users/rhatcher/my_genie_xsec/modified/work/freenuc:
stage 1:  have 12, missing 0
SUCCESS: stage 1

checking stage 2 in /minos/data/users/rhatcher/my_genie_xsec/modified/work/element:
stage 2: have gxspl-freenuc.xml
SUCCESS: stage 2

checking stage 3 in /minos/data/users/rhatcher/my_genie_xsec/modified/work/element:
stage 3:  have 30, missing 0
SUCCESS: stage 3

checking stage 4 in /minos/data/users/rhatcher/my_genie_xsec/modified:
stage 4: have gxspl-NuMIbig.xml
SUCCESS: stage 4

checking stage 5 in /minos/data/users/rhatcher/my_genie_xsec/modified:
stage 5: have gxspl-NuMIsmall.xml
SUCCESS: stage 5

Use in the ART framework

To use under the ART framework one must tell GENIE the name of the spline file and where to find it (and any UserPhysicsOptions.xml file used to make the spline files). In the job .fcl file one can override the defaults for physics.producers.generator along the lines of:

physics.producers.generator.Environment: [
   "GSPLOAD",     "gxspl-NuMIsmall.xml",    # default name of newly created spline file
#  set GXMLPATH so that GENIE knows alternative locations where to find XML files
#  this will allow GENIE to find the new xsec file AND any modified UserPhysicsOptions.xml
#  self-consistency between the two is IMPORTANT
   "GXMLPATH",    "/minos/data/users/rhatcher/my_genie_xsec/modified",  
   "GEVGL",       "Default",
   "GPRODMODE",   "YES",      # supress lots of GENIE messages
#  "GMSGCONF",    "EventGenerator/MyGenieMessengerVerbose.xml",  # override log mesg levels
   "GMSGLAYOUT",  "SIMPLE"  # [BASIC] or timestamp-less SIMPLE, GENIE log4cpp message format
# last entry doesn't have trailing comma!


$ /grid/fermiapp/nusoft/generate_genie_xsec/ -h
  /grid/fermiapp/nusoft/generate_genie_xsec/ -o outpath -s genie_setup [-S gsetup_args]
  Required Args:
    -o <out/path>    output directory for scripts and results:
                        relative to /<expt>/[app|data|ana]/users/<user>
                        subdirectory used for intermediate work products
                        NOTE: it must be *only* the subdir portion of the path
    -s <genie_setup>  script, when sourced, that will setup genie
                        and auxillary packages (e.g. ROOT, log4cpp, LHAPDF)
                        must define  and add to PATH & LD_LIBRARY_PATH
                        special case:  "nova" or "minos" 
    -S <gsetup_args>  arguments to genie_setup script  (e.g. "-r S12.05.31")
                        needs to be quoted if contains a space
  Optional Args:
    -k <nknots>       number of knots to use in splines
    -e <emin>         lowest neutrino energy (not supported by gmkspl)
    -E <emax>         maximum neutrino energy
    -v                increase verbosity level
    -X <XMLpath>      GXMLPATH for GENIE to use for alternative XML files
                        if you are using this to override GENIE parameters
                        then it might be wise to put a copy of the files
                        in the -o directory for reference, and list
                        the same path for this flag.
    -A <apppath>      where this code is installed
    -J <jobsub>       name of condor job submission command
    -G <group>        expt ("minos",  "nova", etc) for BlueArc   
    -U <user>         override "whoami" results

Using spline file from a UPS product

setting up genie_xsec

Making a UPS product out of the spline files

install new genie_xsec