Project

General

Profile

FLUKA is a general purpose tool for calculations of particle transport and interactions with matter, similar to GEANT3 or Geant4. It is available for free to HEP, but under a highly restrictive license . The code is not open source and the bulk of the program comes pre-compiled (currently Linux 32-bit w/ gcc3+g77, and 64-bit w/ gcc4+gfortran). Each version of the library also comes with a "timebomb" that prevents the code from running after a pre-set date; MINOS/NOvA have asked for and received a patch file that pushes that date into the future (which is why we can still run 2008.3d), but it is a reoccurring issue.

FLUGG is a tool that add on to FLUKA to interface to Geant4 geometry.

How to build the FLUGG code

Have you checked that you have permission to access the repository?

Building the proto2-numix branch flugg executable:

It is probably best if one has not already setup an experiment specific environment that might conflict with the necessary environment.

The following assumes you will be working in a directory on an app disk (needs to be mounted executable), say /$EXPT/app/users/$USER/flugg_work_area where $EXPT would be nova or minos or whatever appropriate. Under that directory are going to be one directory that holds the proper FLUGG code, as well as the NuMI-specific g4numi and g4numi_flugg.

Because of the way the upgrade from MINOS-era ("le") to NOvA-era ("mn") geometry was implemented it is not possible to build a single executable that will make a run-time selection. The code can only be built for either on or the other

Initial checkout and directory creation:

  export EXPT=nova

  export FPKGS=/$EXPT/app/users/$USER/flugg_work_area
  mkdir -p $FPKGS
  cd $FPKGS

  export NUMI_BEAM_SIM_CVS=:ext:p-numi-beam-sim@cdcvs.fnal.gov/cvs/projects/numi-beam-sim
  cvs -d${NUMI_BEAM_SIM_CVS} checkout -r proto2-numix flugg_scripts    # for now use this branch

One can then use the create_flugg_version.sh script to build a consistent numi flugg setup.

  export FLUPRO=/path/to/where/FLUKA/is/installed
  flugg_scripts/create_flugg_version.sh -h  # get help
  # build a version named myVersion_{TGT} under $FPKGS
  # for the "nova" era (i.e. "mn" medium energy, NOvA target)
  # (alternative is "minos" era w/ "le" as TGT)
  # so the final version name is "myVersion_mn" 
  # use defaults for -f (2009_3) and -F (latest non-timebombed version) 

  flugg_scripts/create_flugg_version.sh -b $FPKGS -n myVersionBase -e nova 
  [...progress information...]

This will create a g4numi_flugg release by fetching FLUGG, and checking out g4numi, g4numi_flugg from the repository, configuring the code (including necessary selection/editing of files), and building the executable.

  $ cd ${FPKGS}/myVersionBase_mn 
  $ ls -1 
build.info             # basic configuration info
build.log              # explicit build details
FLUGG                  # unpacked FLUGG + config script + executable
flugg_2009_4.tar.gz    # downloaded FLUGG source
g4numi                 # NuMI code from CVS + appropriate symlinks
g4numi_flugg           # NuMI code from CVS + appropriate symlinks
scripts                # symlink to g4numi_flugg/scripts for convenience

  $ find . -type l  # what symlinks are made in the process
./g4numi/include/NumiDataInput.hh
./g4numi/src/NumiDetectorConstruction.cc
./g4numi/src/NumiTarget.cc
./g4numi/src/NumiTargetHall.cc
./g4numi/src/NumiHorn1.cc
./g4numi/src/NumiDataInput.cc
./g4numi_flugg/include/NumiDataInput.hh
./g4numi_flugg/include/NumiDetectorConstruction.hh
./g4numi_flugg/include/NumiHornSpiderSupport.hh
./g4numi_flugg/include/NumiMagneticField.hh
./g4numi_flugg/for/fluscw.f
./g4numi_flugg/for/mgdraw.f
./g4numi_flugg/src/NumiBaffle.cc
./g4numi_flugg/src/NumiDetectorConstruction.cc
./g4numi_flugg/src/NumiHorn2.cc
./g4numi_flugg/src/NumiMaterials.cc
./g4numi_flugg/src/NumiTargetHall.cc
./g4numi_flugg/src/NumiDataInput.cc
./g4numi_flugg/src/NumiHadronAbsorber.cc
./g4numi_flugg/src/NumiHornSpiderSupport.cc
./g4numi_flugg/src/NumiSecMonitors.cc
./g4numi_flugg/src/NumiDecayPipe.cc
./g4numi_flugg/src/NumiHorn1.cc
./g4numi_flugg/src/NumiMagneticField.cc
./g4numi_flugg/src/NumiTarget.cc
./scripts

# a "cvs diff" would (generally) show that either mgdraw.f.nova or mgdraw.f.minos had been modified

The -h or --help flags give help info:

$ flugg_scripts/create_flugg_version.sh -h

create_flugg_version.sh -b BASEPATH -n NAME -e ERA -t NUMITAG -f FLUGGVER 

Create a g4numi_flugg release by fetching FLUGG, and checking out
g4numi, g4numi_flugg from the repository, configuring the code (including
necessary selection/editing of files), and building the executable.

   -h | --help                 print this message and quite
   -v                          increase verbosity
   -b | --basepath  BASEPATH   base installation 
                               [/minos/app/user/rhatcher/g4numi_flugg_build]
   -n | --vername   VERNAME    version name under base [myversion]
                               will have '_le' or '_mn' appended
   -e | --era       ERA        minos / nova era (ie. le or mn beam) [minos]
   -d | --detloc    DETLOC     detector location ( "minos" or "nova" )
                               for h10 style ntuple
   -t | --numitag   NUMITAG    cvs tag for numi-x cvs checkout [proto2-numix]
   -r | --remake               allow "remake" 
                               (no checkout, no refetch of flugg tarball)
   -2 | --cout2cerr=true|false convert NuMI code use of G4cout to G4cerr
                               FLUGG mashes lots of things into g4numi001.log
                               that overwrite each other, losing output
                               attempt to alleviate that [false]
   -g | --gcc       WHICH_GCC  setup what gcc [v3_4_3]
   -f | --fluggver  FLUGGVER   flugg version to fetch [2009_3]
                               e.g. 2009_3 or 2009_4
   -F | --flukapro  FLUPRO     fluka installation path
                               [/grid/fermiapp/nusoft/products/prd/fluka/fluka2011.2b.6/gcc_3_4_3_g77]
                               special cases: see below
   -G | --g4install G4INSTALL  geant4 installation 
                               [/grid/fermiapp/nusoft/products/prd/geant4/4.9.3/Linux+2.6-GCC_3_4]
   -C | --clhep     CLHEPDIR   CLHEP installation 
                               [/grid/fermiapp/nusoft/products/prd/clhep/2.0.3.2/Linux32bit+2.6]
   -H | --cernlib   CERNLIB    CERNLIB installation (for ascii2hbook_beam)
                               [/grid/fermiapp/minos/products/prd/cern/2004/Linux-2-6/lib/] 

--flukapro=2008_3d = /grid/fermiapp/nusoft/products/prd/fluka/2008.3d/Linux+2.6-GCC_3_4
--flukapro=2011_2b = /grid/fermiapp/nusoft/products/prd/fluka/fluka2011.2b.6/gcc_3_4_3_g77

Modifying the code and rebuilding

If you need to modify the code then a rebuild can be accomplished by invoking the exact same command (i.e. same flags) with the addition of --remake. The code tries to avoid introducing confusion by preventing some mis-matches but isn't guaranteed to catch them all (maybe in time it can be made smarter).

Tags

tag description
proto2-numix "HEAD" branch of NuMI-X flugg development
proto2-numix-initial state of the code when first branched for numix
numix-blackbird target and horn2 position adjustments, exponential horn current,
run# bug fix, Nimpwt fix by using upstream fake volume
numix-crow not yet ...

Running the code

There are two scripts that work in conjunction to run the flugg code. It is assumed that both will be in the same location. There are standard versions in the build release, but one could also checkout the flugg_scripts CVS package and modify them locally (i.e. independent of the built release) in which case you must additionally tell the script where FLUGG is built.

export PERIOD=11  # nova era
export LOWTH="true" 
export NPOTS=5000

export JOBNUM=42
nohup $FPKGS/myVersionBase_mn/g4numi_flugg/scripts/run_flugg_grid.sh \
   -e $NPOTS -r $JOBNUM  -p $PERIOD -l $LOWTH --peanut=true \
   --dk2nu --targfile=true \
   > run_short_mn_${JOBNUM}.log 2>&1 &

The run_flugg_grid.sh script is grid compatible. Be aware that the default expected lifetime of a job at submission is 8h. Currently a typical 500K POTs FLUGG job takes about 12h (e.g. 12h = 12*3600 = 43200s). To avoid having jobs being placed on hold, one should explicitly define the expected lifetime of a job at submission.

source /grid/fermiapp/products/common/etc/setups.sh
setup jobsub_client
export JOBSUB_GROUP=nova

export PERIOD=11  # nova era
export LOWTH="true" 
export NPOTS=500000
export NJOBS=1000
export OUTPUTDIR=/pfns/nova/scratch/users/$USER
export EXPLIFE=64800

jobsub_submit -g --group ${JOBSUB_GROUP} -N ${NJOBS} --expected-lifetime=${EXPLIFE}\
   file://$FPKGS/myVersionBase_mn/g4numi_flugg/scripts/run_flugg_grid.sh \
   -e $NPOTS -r 0  -p $PERIOD -l $LOWTH --peanut=true \
   --dk2nu --targfile=true -o ${OUTPUTDIR}

Submits 1000 jobs of 500K POTs with run numbers 0 to 999.

Again, help can be obtained using the -h or --help flags:

$ ${BASEPATH}/${VERNAME}_mn/run_flugg_grid.sh -h

run_flugg_grid.sh -p RUNPERIOD -j JOB -e EVTS [ -F FLUGGINSTALL ]
      [-o OUTPATH] [ --fakeinfo "FAKEINFO" ]

Run a FLUGG job, copy files back
   -p | --period=RUNPERIOD    run period (e.g. 3,  nova=11 ...) [11]
   -j | --job=JOBNUMBASE      first job # [0=0+(unset)]
                              if on cluster JOB # = JOBNUMBASE + PROCESS
   -r | --run=JOBNUMBASE      synonym for job
   -e | --events=EVTS         # events (protons-on-target) [500000]
   -i | --horncur=CURRENT     horn current [200]
   -z | --targetz=TARGETZ     target pull-back [0]
   -s | --stepl=STEPL         max step length (needs 3 char w/ decimal point)
                              [1.0]
   -S | --stepm=STEPM         min step (>0) or boundary uncertainty (<0) (cm) 
                              [-0.005]
   -b | --beamspot=BSSIGMA    beam spot sigma (mm) currently NOvA-era ONLY 
                              1 or 2 values ala:  "1.3" or "1.3x1.1" 
   -B | --beampos=BSPOT       beam spot position (mm) use "<x>,<y>[,<z>]" 
                              z optional; default = 0,0,-6000  
   -T | --targfile=true|false enable target file production [false]
   -A | --ancfile=true|false  enable ancestor list production [true]
        --printdelta          add delta ray interactions in the Ancestor List
   -l | --lowth=true|false    enable low threshold [true]
   -P | --peanut=true|false   enable PEANUT for all energies [true]
   -u | --unique=UNIQUENESS   string to make file unique [""]
        --special=SPECIAL     "" or _airhrn _fins78 _fins6789 _mipp _mudecay 
                              [""]
   -t | --test                run wrapper as test, but don't invoke subscript
        --hbook               generate hbook .ntp file [false]
        --keepdat             keep g4numi001_flux.dat file
        --keepanc             keep g4numi001_ancestors.dat file
        --altrfluka[=false]   turn on/off alt "rfluka" that better merges
                              sys-err into sys-out [true]
                              (still might be mangled, but early output
                              should no longer get completely lost)
   -D | --dk2nu[=false]       copy/convert file into dk2nu format 
                              use "version%qualifiers" 
                              [v01_02_00%e5:debug]  true=v01_02_00%e5:debug
   -F | -f | --flugg=FLUGGINSTALL  where FLUGG installed, 
                              should have subdir {bin|lib}/Linux-g++
                              if not given, 2-3 dir up + FLUGG from this script
                    []
   -o | --output=OUTPATH      top of hierarchy where output will go
                    [/nova/data/condor-tmp/rhatcher/FLUGG]
   -R | --rootsys=ROOTSYS     where ROOT is installed
                    [/grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/v5-28-00]

        --skindepth=0.77      use exponential horn skin depth of 0.77cm
                              [""] ""=use linear model
        --fakeinfo=FAKEINFO   configuration info for fake volume ["DEFAULT"]
                              use " XPOS YPOS ZPOS R DZ [MAXFAKEOUT] " units=cm
                              use "DEFAULT" to have it on, but use default
                              one word alternatives: "OFF" or "DISPLACED" 

Dealing with Failures

Jobs fail for a number of reasons; some due to the batch system, some due exceptions in the FLUKA code. We only want to keep those files that ran to completion. Results from individual jobs come back as directories called RunXXXX collected under an umbrella directory. By using the read_the_whirlwind.sh script one can sort those results, weed out the bad files and make a directory of symlinks to all the root files, dk2nu files and/or target files.

cvs -d ${NUMI_BEAM_SIM_CVS} checkout -p -r proto2-numix g4numi_flugg/scripts/reap_the_whirlwind.sh > reap_the_whirlwind.sh
chmod +x reap_the_whirlwind.sh

bash reap_the_whirlwind.sh <directory-of-runs>

In the example below Run00002 failed in some identifiable way, and job 0001 was run without the option to produce the target file. Then the resulting directory structure might look like:

flugg_mn000z200i_rp11_lowth_pnut_f112c0f093bbird
  Run0000
    flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0000.{root|dk2nu.root|log|out|err}
    target0000.root
    <assorted-other-files>
  Run0001
    flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0001.{root|dk2nu.root|log|out|err}
    <assorted-other-files>
  Run0002.BAD
    flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0002.{root|dk2nu.root|log|out|err}
    <assorted-other-files>
  Run0003
    flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0003.{root|dk2nu.root|log|out|err}
    target0003.root
    <assorted-other-files>
  ...
  dk2nu
    symlink to ../Run0000/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0000.dk2nu.root
    symlink to ../Run0001/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0001.dk2nu.root
    symlink to ../Run0003/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0003.dk2nu.root
  root
    symlink to ../Run0000/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0000.root
    symlink to ../Run0001/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0001.root
    symlink to ../Run0003/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0003.root
  target
    symlink to ../Run0000/target0000.root
    symlink to ../Run0003/target0003.root

Generally one only need supply the directory to be sorted, but there are some options. The --no-target option is useful if none of the jobs were run to generate target files, so it doesn't whinge so much.

reap_the_whirlwind.sh [options] <top-dir>

Purpose:  Rummange around under <top-dir> for directories names Run????.
          Determine if that particular flugg run failed (looking in err
          logs); rename the director as Run????.Bad if so deemed.
          Then make subdirectories "root" and "dk2nu" and populate
          them with symlinks to remaining good files.

 --revert      restore all Run????.Bad directories to Run????
 --no-bad      don't try to classify runs (use w/ --revert to start fresh)
 --no-flugg    don't make flugg links
 --no-dk2nu    don't make dk2nu links
 --no-target   don't make target links
 --no-refresh  don't wipe flugg/dk2nu before starting
 --maxlinks=   limit # of symlinks (to equalize samples) [10000]
 --agemin=     minimum age of directory in minutes [10]

FLUKA FLUGG combinations known to work

gcc 3.4.3 32-bit flugg 2009_3 flugg 2009_4
fluka 2008.3d known to run
fluka 2011.2b.6 runs builds (MAGNEW log errors)

A word about compilers and fluka/flugg

FLUGG currently (2014-08) can NOT be run 64-bit because it attempts to shove a pointer (which would be 64-bit) into a G4int (which is 32-bit).

Peanut

About the PEANUT option:

http://www.fluka.org/web_archive/earchive/new-fluka-discuss/2311.html note of possible interest

This is the line it adds:

if [ "$PEANUT" == "true" ]; then
echo "Special PEANUT run, model used for all energies." 
echo "PHYSICS    1000.     1000.     1000.     1000.     1000.     1000.    PEATHRES" >> ${inp}
fi

Yet older instructions