Running NOvA Simulations

There are three different ways to create events for NOvA:

  1. GENIE - neutrino interaction generation
  2. CRY - cosmic ray generation
  3. Single - creation of predefined particles at given energies, directions in the detectors

The EventGenerator and EventGeneratorBase packages contain a module for each of these, as well as fcl files defining the jobs to run each type of event generation.

Subsequent to the initial event generation are three additional steps. The initial list of particles from the event generator must be propagated through the geometry undergoing physics processes (e.g. multiple scattering, energy loss mechanisms, annihilation, decay, etc.). It is in this stage that a list of true energy losses are recorded for the sensitive volumes (i.e. the cells of liquid scintillator). The final two phases involve taking the energy depositions (hits) and simulating the readout system, including the light propagation, photodetectors and electronics to create raw data structures (digits) of the same form as produced in the real detector.

This page contains instructions for producing a file from the event generation stage through the raw digit simulation. Instructions for reconstructing the produced events are elsewhere. The job level .fcl file will have a section that looks similar to:

   generator: @local::genie_simpleflux_ndos
   geantgen:  @local::standard_geant4
   photrans:  @local::standard_photrans
   daq:       @local::standard_rsim
   occupied:  @local::standard_occupied_filter

Running with Fixed Seeds

Sometimes when debugging, one wants to be able to generate the same series of events over and over again. To do this, one needs to fix four random number generator seeds. GENIE has a seed, as well as geant, photon transport, and daq. GENIE will accept a seed up to 10 digits (not all are unique), the others will accept 9 digit numbers (again, not all are necessarily unique). You can set these in your job fcl file by adding lines at the end like:

     physics.producers.generator.RandomSeed: "600980270" 
     physics.producers.geantgen.Seed: "243968665" 
     physics.producers.photrans.Seed: "818716369" 
     physics.producers.daq.Seed: "269041435" 

where obviously you can set the numbers to whatever you want the seeds to be.


Please see the description in the NuTools wiki.

Initial Event Generation

At a minimum the event generator must produce a list of particles along with their 4-momenta and positions that represent some kind of "event". Some generators (e.g. GENIE) record additional information about the initial state (neutrino and nucleus info), what process occurred (basic neutrino kinematics), and actions that took place within the original nuclear environment (e.g. inter-nuclear scattering).

Neutrino Interaction Generation with GENIE

partially revised 2011-09-23

To create a file containing neutrino interactions for each event, do

% nova -c prodgenie.fcl -n nevents -o outfile.root -T outhist.root

configuring GENIE

For the GENIE simulation, the EventGenerator/GENIEGen.fcl file contains some parameters that a person may be likely to change.
Information on what can be changed can be found at GENIEHelper

Additions to master the fcl file might look something like (don't use these, they're just examples):

physics.producers.generator.MixerConfig: "map 12:14 14:12 -12:-14 -14:-12" 
physics.producers.generator.FluxUpstreamZ: -100000 
physics.producers.generator.FiducialCut:  "rockbox:(-780,-780,20)(780,780,6500),0,800,1.0e-5,1.10" 
physics.producers.generator.FiducialCut: "mbox:-200,-200,-50,200,200,1600"     # NearDet to exclude rock inside vDetEnclosure
physics.producers.generator.GeomScan:     "flux: 10000 1.2 1" 
#physics.producers.generator.GeomScan:     "file: Geometry/maxpathlengths/NOvA_FAR-ROCK.maxpl.xml" 
##physics.producers.generator.DebugFlags:   4294967295 # 0xFFFFFFFF
physics.producers.generator.ZCutOff:      100000
physics.producers.generator.Environment: [
     "GSPLOAD",    "gxspl-NUMI-R2.6.0.xml",
     "GEVGL",      "Default",
     "GPRODMODE",  "YES",      # supress lots of GENIE messages
     "GMSGLAYOUT", "SIMPLE",  # [BASIC] or timestamp-less SIMPLE
     "GMSGCONF",   "MyGenieMsgVerbose.xml:EventGenerator/MyGenieMessengerVerbose.xml",  # override log mesg levels
     "GXMLPATH",   "/path/to/my/xml/files:/another/path" 

A few words about flux files in general

By default GENIEHelper will make copies of the flux files it is going to use into a local directory. This is the desired behavior in the case of running on the grid nodes. It is likely problematic on the interactive GPVMs (as, by default, they will end up in /var/tmp which isn't that big). Put the following at the bottom of your fcl file:

 physics.producers.generator.FluxCopyMethod: "DIRECT" 
in order to read the files directly (only available where PNFS directory is mounted locally).
See nugen Flux File Handling.

A few words about GSimpleNtpFlux files

NOvA generally uses GSimpleNtpFlux files. These are created from flugg or g4numi beam simulation output and are evaluated (a CPU costly step) for a particular detector location and possibly with different window sizes depending on one's interest in generating event in the detector only or also the surrounding "rock". NOvA's flux files are located at /nova/data/flux/gsimple/; currently the most up-to-date versions are in the sub-directory nova_v03. Below that the hierarchical structure reflects detector position (+window size), beam line configuration (target+horn) and horn current, ala: /nova/data/flux/gsimple/nova_v03/ndrock/mn/fhc/

The GENIEHelper can be told to use a different set of flux files by a line like:

physics.producers.generator.FluxFiles:  [ "flux/gsimple/nova_v03/ndos/mn/fhc/*.root" ]

It is imperative that one not mix up detector configuration and incompatible flux files (i.e. don't generate with ND geometry and NDOS flux).

Instructions on dumping some useful information about a file can be found at:
And more generally (including generation of "gsimple" files) at:

Generating Neutrino Events in the Rock

The default NOvA configuration of GENIE generates events only in the "vDetEnclosure" volume. To generate events in the rock in an efficient and unbaised manner then several adjustments must be made. These are described on this page.

Generating Using an Alternative GENIE Version

The NOvA code can be built against a version of GENIE other than the default. But special care must be taken to ensure that the build is consistent. These are described on this page

Cosmic Ray Generation with CRY

To create a file containing cosmic ray interactions for each event, do

% nova -c prodcosmics.fcl

CRY cosmic ray generator simulates all kinds of cosmic particles at the sea level. We then take them and project them to the end of the Geant4's "World Volume" of the NOvA detector. Tracking all cosmic particles through Geant4 would take a long time, especially for the far detector as the World volume there is bigger. So, we look if the particle would intersect some "Big Box" around the detector. It is defined as the box where an additional X value of centimeters is added to each of the dimensions of the detector height, width and length. Typically we take X=1500 cm, but it is a configurable parameter in Geometry.fcl file.

Single Particle Generation

You can make files where each event contains a predetermined set of particles by doing

% nova -c prodsingle.fcl -n nevents -o outfile.root -T outhist.root --nskip nskippedevents

You can edit parameters related to the run number, number of events to generate, and detector geometry to use in prodsingle.fcl located in the EventGenerator package. You can edit parameters related to the initial particle momentum, vertex position, direction, and flavor in SingleGen.fcl in the EventGenerator package. The events can either contain exactly one particle per event or a set of several particles depending on the configuration you use. See EventGenerator/prodsingle.fcl for more details.

Producing FLSHits using the Geant4 Simulation

Geant4 is used to propagate particles that come out of the generator (genie, cry, etc.) through the geometry taking care of physics processes such as energy loss, multiple scattering, hadronic & electromagnetic interactions and decay. Energy losses (ionization) in "active" volumes that could generate detectable signals are recorded at this stage. In NOvA the data structure FLSHit records this information.

Geant4 is normally run as part of the event generation sequence and no longer done as a separate job process.

NOvA uses Geant4 as an external package that is relied upon to do the desired work with known, stable interfaces. Some FNAL Geant4 efforts involve customization of the physics list which controls the physics processes, especially in regard to the hadronic interactions.

The physics list used by Geant4 determines how particles are propagated and interact. There are a number of pre-defined variations; NOvA, by default, uses QGSP_BERT. This can be overridden in the fcl file by a line of the form:

physics.producers.geantgen.G4AlgPSet.G4PhysListName: "QGSP_BERT_HP" 

The physics lists currently (subject to change) known to Geant4 (v4_9_4_p01) G4PhysListFactory are:

  • LBE
  • LHEP
  • QBBC
  • QGSP
  • Shielding

some of which are described on

A current enumeration of registered physics lists can be obtained by using a nonsensical name in a test run.

Adding and configuring additional physics processes to a physics list

The G4Helper also provides a hook for adding in additional physics processes, in particular monopoles, and configuring them:

physics.producers.geantgen.G4AlgPSet.G4PhysListName: " QGSP_BERT ;
   ( /monopole/setup 1. 0. 1.0e5 GeV ,
   ( /monopole/magCharge 1 ,
     /monopole/elCharge 0 ) " 

The first field (split using a semi-colon as a separator) is always the base physics list. Subsequent fields start with a physics process derived from G4VPhysicsConstructor and registered with the G4Base/G4PhysicsProcessFactorySingleton. Comma separated commands given in ()'s are passed via the G4UI (user interface) to configure the process at the appropriate time.

The current setting can be found here:

Warning Message

If you see the message in the log:

      **** COMMAND NOT FOUND </...> *****
      **** Batch is interrupted!! *****
don't panic. These messages are completely innocuous for NOvA, and are an artifact of our choice of physics list (_HP = high precision neutrons) which doesn't have a "neutron killer" component to configure with this command (other physics lists do, so we want them to not kill all the neutrons on the first step).

This is generated by the lines in:

# Set time after which to kill neutrons; pick a long time (needs units).
# One can also set a minimum energy for the neutron (kill the neutron
# if its KineticEnergy falls below the set value).
# This command is valid for most (but *not* all) physics lists,
# in particular LHEP and the _HP lists don't support it and it will cause a 
#      **** COMMAND NOT FOUND </...> *****
#      **** Batch is interrupted!! *****
# message when run on such a list.
/physics_engine/neutron/timeLimit 515000.0 ns
/physics_engine/neutron/energyLimit 0.0 GeV
which is harmless as long as no relevant commands are in g4nova.mac after this point.

Configuring Track ID's

In standard running, we don't keep every individual particle that is made. Sometimes, if a daughter is made from an EM shower (that is, has a process id of: conv, LowEnConversion, Pair, compt, Compt, Brem, phot, Photo, Ion, or annihil) we instead associate the activity with the mother particle. However, one might with to have this full set of information at times, especially when debugging. To do this, one should change their prodgenie.fcl file (or whatever file one is using to produce their events). Then the line:

   geantgen:  @local::standard_geant4

should be changed to:

   geantgen:  @local::manyParticles_geant4

This is simply a different configuration of geant where we don't do the step of putting EM daughters with their mother particle.

Additional G4 UserActions

Classes that derive from G4Base/UserAction can be added to the list of actions called during G4 processing; such classes must self-register with the UserActionFactory to be used in the NOvA framework. Examples can be found at G4Base/ExampleAction and g4nova/RockCutterAction. They can be added to the list of actions taken and configured by entries in the fcl file along the lines of:

physics.producers.geantgen.G4AlgPSet.AddedUserActions: [ "g4n::RockCutterAction" 
physics.producers.geantgen.G4AlgPSet.RockCutterAction: { 
  Verbose: 42
  X0:0       #cm
  X_max: 304.8   #cm
  X_min: -1620
  Y_max: 407.9
  Y_min: -178.9
  Z_max: 1800
  Z_min: -840

Notice: The 3 parameters X0, Y0 and Z0 are the center coordinates of the nd, and the later 6 parameters defines a grand rockbox. If you do not specify them in your fcl file, the default values are listed above. Once you want it to work for fd geo, you ought to modify these parameters.

Having Geant4 check the geometry for overlaps

By default, NOvA does not have Geant4 to check for geometry overlaps in each-and-every job. That is an unnecessary step during production and an annoying delay during interactive use. But any changes to the geometry or Geant4 version should include a short job that enables this check for each of the geometries.

physics.producers.geantgen.G4AlgPSet.G4OverlapCheck = true
physics.producers.geantgen.G4AlgPSet.G4ValidateGDMLSchema = true

Modifying particle lifetimes in Geant4

Danger (ordinary mortals should ignore this section)

In very special circumstances (e.g. to generate an enhanced set of muons decaying in flight) one might want to modify the lifetime of particles in Geant4. This can be done using the following:

# configure Geant4 to modify the lifetime of mu- and mu+
physics.producers.geantgen.G4AlgPSet.OverrideLifetimePDG:  [ 13, -13 ]
physics.producers.geantgen.G4AlgPSet.OverrideLifetimeNS:   [ 2196.98, 2196.98 ]  # ns

Producing Raw Digits Using the Geant4 Energy Depositions

Needs expansion!

This normally occurs as part of the job path for all three cases.

Verifying the Monte Carlo is Working

The best way to check that the file you just made is to use the source:MCCheckout package. To run a check-out job, do

% nova -c mccheckoutjob.fcl -s detsim.root -o out.root

Where out.root is the file containing the histograms made. There are 4 modules that one can run using MCCheckOut:

  • NeutrinoAna: produces histograms related to the information for generated neutrino interactions
  • CosmicAna: produces histograms related to the information for generated cosmic ray interactions
  • DetSimAna: produces histograms related to the information for the Geant4 simulation
  • DetAna: produces histograms related to the information for the raw digit simulation

The modules to run are set by editing the MCCheckout/mccheckoutjob.fcl file and setting which modules to use. One would likely never use the CosmicAna and NeutrinoAna modules at the same time. The output file will have a folder in it named for each module that will contain the produced histograms.