- Table of contents
- How to build the FLUGG code
FLUKA is a general purpose tool for calculations of particle transport and interactions with matter, similar to GEANT3 or Geant4. It is available for free to HEP, but under a highly restrictive license . The code is not open source and the bulk of the program comes pre-compiled (currently Linux 32-bit w/ gcc3+g77, and 64-bit w/ gcc4+gfortran). Each version of the library also comes with a "timebomb" that prevents the code from running after a pre-set date; MINOS/NOvA have asked for and received a patch file that pushes that date into the future (which is why we can still run 2008.3d), but it is a reoccurring issue.
FLUGG is a tool that add on to FLUKA to interface to Geant4 geometry.
How to build the FLUGG code¶
Have you checked that you have permission to access the repository?
Building the proto2-numix branch flugg executable:¶
It is probably best if one has not already setup an experiment specific environment that might conflict with the necessary environment.
The following assumes you will be working in a directory on an app disk (needs to be mounted executable), say /$EXPT/app/users/$USER/flugg_work_area where $EXPT would be nova or minos or whatever appropriate. Under that directory are going to be one directory that holds the proper FLUGG code, as well as the NuMI-specific g4numi and g4numi_flugg.
Because of the way the upgrade from MINOS-era ("le") to NOvA-era ("mn") geometry was implemented it is not possible to build a single executable that will make a run-time selection. The code can only be built for either on or the other
Initial checkout and directory creation:
export EXPT=nova
export FPKGS=/$EXPT/app/users/$USER/flugg_work_area
mkdir -p $FPKGS
cd $FPKGS
export NUMI_BEAM_SIM_CVS=:ext:p-numi-beam-sim@cdcvs.fnal.gov/cvs/projects/numi-beam-sim
cvs -d${NUMI_BEAM_SIM_CVS} checkout -r proto2-numix flugg_scripts # for now use this branch
One can then use the create_flugg_version.sh script to build a consistent numi flugg setup.
export FLUPRO=/path/to/where/FLUKA/is/installed
flugg_scripts/create_flugg_version.sh -h # get help
# build a version named myVersion_{TGT} under $FPKGS
# for the "nova" era (i.e. "mn" medium energy, NOvA target)
# (alternative is "minos" era w/ "le" as TGT)
# so the final version name is "myVersion_mn"
# use defaults for -f (2009_3) and -F (latest non-timebombed version)
flugg_scripts/create_flugg_version.sh -b $FPKGS -n myVersionBase -e nova
[...progress information...]
This will create a g4numi_flugg release by fetching FLUGG, and checking out g4numi, g4numi_flugg from the repository, configuring the code (including necessary selection/editing of files), and building the executable.
$ cd ${FPKGS}/myVersionBase_mn
$ ls -1
build.info # basic configuration info
build.log # explicit build details
FLUGG # unpacked FLUGG + config script + executable
flugg_2009_4.tar.gz # downloaded FLUGG source
g4numi # NuMI code from CVS + appropriate symlinks
g4numi_flugg # NuMI code from CVS + appropriate symlinks
scripts # symlink to g4numi_flugg/scripts for convenience
$ find . -type l # what symlinks are made in the process
./g4numi/include/NumiDataInput.hh
./g4numi/src/NumiDetectorConstruction.cc
./g4numi/src/NumiTarget.cc
./g4numi/src/NumiTargetHall.cc
./g4numi/src/NumiHorn1.cc
./g4numi/src/NumiDataInput.cc
./g4numi_flugg/include/NumiDataInput.hh
./g4numi_flugg/include/NumiDetectorConstruction.hh
./g4numi_flugg/include/NumiHornSpiderSupport.hh
./g4numi_flugg/include/NumiMagneticField.hh
./g4numi_flugg/for/fluscw.f
./g4numi_flugg/for/mgdraw.f
./g4numi_flugg/src/NumiBaffle.cc
./g4numi_flugg/src/NumiDetectorConstruction.cc
./g4numi_flugg/src/NumiHorn2.cc
./g4numi_flugg/src/NumiMaterials.cc
./g4numi_flugg/src/NumiTargetHall.cc
./g4numi_flugg/src/NumiDataInput.cc
./g4numi_flugg/src/NumiHadronAbsorber.cc
./g4numi_flugg/src/NumiHornSpiderSupport.cc
./g4numi_flugg/src/NumiSecMonitors.cc
./g4numi_flugg/src/NumiDecayPipe.cc
./g4numi_flugg/src/NumiHorn1.cc
./g4numi_flugg/src/NumiMagneticField.cc
./g4numi_flugg/src/NumiTarget.cc
./scripts
# a "cvs diff" would (generally) show that either mgdraw.f.nova or mgdraw.f.minos had been modified
The -h or --help flags give help info:
$ flugg_scripts/create_flugg_version.sh -h
create_flugg_version.sh -b BASEPATH -n NAME -e ERA -t NUMITAG -f FLUGGVER
Create a g4numi_flugg release by fetching FLUGG, and checking out
g4numi, g4numi_flugg from the repository, configuring the code (including
necessary selection/editing of files), and building the executable.
-h | --help print this message and quite
-v increase verbosity
-b | --basepath BASEPATH base installation
[/minos/app/user/rhatcher/g4numi_flugg_build]
-n | --vername VERNAME version name under base [myversion]
will have '_le' or '_mn' appended
-e | --era ERA minos / nova era (ie. le or mn beam) [minos]
-d | --detloc DETLOC detector location ( "minos" or "nova" )
for h10 style ntuple
-t | --numitag NUMITAG cvs tag for numi-x cvs checkout [proto2-numix]
-r | --remake allow "remake"
(no checkout, no refetch of flugg tarball)
-2 | --cout2cerr=true|false convert NuMI code use of G4cout to G4cerr
FLUGG mashes lots of things into g4numi001.log
that overwrite each other, losing output
attempt to alleviate that [false]
-g | --gcc WHICH_GCC setup what gcc [v3_4_3]
-f | --fluggver FLUGGVER flugg version to fetch [2009_3]
e.g. 2009_3 or 2009_4
-F | --flukapro FLUPRO fluka installation path
[/grid/fermiapp/nusoft/products/prd/fluka/fluka2011.2b.6/gcc_3_4_3_g77]
special cases: see below
-G | --g4install G4INSTALL geant4 installation
[/grid/fermiapp/nusoft/products/prd/geant4/4.9.3/Linux+2.6-GCC_3_4]
-C | --clhep CLHEPDIR CLHEP installation
[/grid/fermiapp/nusoft/products/prd/clhep/2.0.3.2/Linux32bit+2.6]
-H | --cernlib CERNLIB CERNLIB installation (for ascii2hbook_beam)
[/grid/fermiapp/minos/products/prd/cern/2004/Linux-2-6/lib/]
--flukapro=2008_3d = /grid/fermiapp/nusoft/products/prd/fluka/2008.3d/Linux+2.6-GCC_3_4
--flukapro=2011_2b = /grid/fermiapp/nusoft/products/prd/fluka/fluka2011.2b.6/gcc_3_4_3_g77
Modifying the code and rebuilding¶
If you need to modify the code then a rebuild can be accomplished by invoking the exact same command (i.e. same flags) with the addition of --remake. The code tries to avoid introducing confusion by preventing some mis-matches but isn't guaranteed to catch them all (maybe in time it can be made smarter).
Tags¶
| tag | description |
|---|---|
| proto2-numix | "HEAD" branch of NuMI-X flugg development |
| proto2-numix-initial | state of the code when first branched for numix |
| numix-blackbird | target and horn2 position adjustments, exponential horn current, run# bug fix, Nimpwt fix by using upstream fake volume |
| numix-crow | not yet ... |
Running the code¶
There are two scripts that work in conjunction to run the flugg code. It is assumed that both will be in the same location. There are standard versions in the build release, but one could also checkout the flugg_scripts CVS package and modify them locally (i.e. independent of the built release) in which case you must additionally tell the script where FLUGG is built.
export PERIOD=11 # nova era
export LOWTH="true"
export NPOTS=5000
export JOBNUM=42
nohup $FPKGS/myVersionBase_mn/g4numi_flugg/scripts/run_flugg_grid.sh \
-e $NPOTS -r $JOBNUM -p $PERIOD -l $LOWTH --peanut=true \
--dk2nu --targfile=true \
> run_short_mn_${JOBNUM}.log 2>&1 &
The run_flugg_grid.sh script is grid compatible. Be aware that the default expected lifetime of a job at submission is 8h. Currently a typical 500K POTs FLUGG job takes about 12h (e.g. 12h = 12*3600 = 43200s). To avoid having jobs being placed on hold, one should explicitly define the expected lifetime of a job at submission.
source /grid/fermiapp/products/common/etc/setups.sh
setup jobsub_client
export JOBSUB_GROUP=nova
export PERIOD=11 # nova era
export LOWTH="true"
export NPOTS=500000
export NJOBS=1000
export OUTPUTDIR=/pfns/nova/scratch/users/$USER
export EXPLIFE=64800
jobsub_submit -g --group ${JOBSUB_GROUP} -N ${NJOBS} --expected-lifetime=${EXPLIFE}\
file://$FPKGS/myVersionBase_mn/g4numi_flugg/scripts/run_flugg_grid.sh \
-e $NPOTS -r 0 -p $PERIOD -l $LOWTH --peanut=true \
--dk2nu --targfile=true -o ${OUTPUTDIR}
Submits 1000 jobs of 500K POTs with run numbers 0 to 999.
Again, help can be obtained using the -h or --help flags:
$ ${BASEPATH}/${VERNAME}_mn/run_flugg_grid.sh -h
run_flugg_grid.sh -p RUNPERIOD -j JOB -e EVTS [ -F FLUGGINSTALL ]
[-o OUTPATH] [ --fakeinfo "FAKEINFO" ]
Run a FLUGG job, copy files back
-p | --period=RUNPERIOD run period (e.g. 3, nova=11 ...) [11]
-j | --job=JOBNUMBASE first job # [0=0+(unset)]
if on cluster JOB # = JOBNUMBASE + PROCESS
-r | --run=JOBNUMBASE synonym for job
-e | --events=EVTS # events (protons-on-target) [500000]
-i | --horncur=CURRENT horn current [200]
-z | --targetz=TARGETZ target pull-back [0]
-s | --stepl=STEPL max step length (needs 3 char w/ decimal point)
[1.0]
-S | --stepm=STEPM min step (>0) or boundary uncertainty (<0) (cm)
[-0.005]
-b | --beamspot=BSSIGMA beam spot sigma (mm) currently NOvA-era ONLY
1 or 2 values ala: "1.3" or "1.3x1.1"
-B | --beampos=BSPOT beam spot position (mm) use "<x>,<y>[,<z>]"
z optional; default = 0,0,-6000
-T | --targfile=true|false enable target file production [false]
-A | --ancfile=true|false enable ancestor list production [true]
--printdelta add delta ray interactions in the Ancestor List
-l | --lowth=true|false enable low threshold [true]
-P | --peanut=true|false enable PEANUT for all energies [true]
-u | --unique=UNIQUENESS string to make file unique [""]
--special=SPECIAL "" or _airhrn _fins78 _fins6789 _mipp _mudecay
[""]
-t | --test run wrapper as test, but don't invoke subscript
--hbook generate hbook .ntp file [false]
--keepdat keep g4numi001_flux.dat file
--keepanc keep g4numi001_ancestors.dat file
--altrfluka[=false] turn on/off alt "rfluka" that better merges
sys-err into sys-out [true]
(still might be mangled, but early output
should no longer get completely lost)
-D | --dk2nu[=false] copy/convert file into dk2nu format
use "version%qualifiers"
[v01_02_00%e5:debug] true=v01_02_00%e5:debug
-F | -f | --flugg=FLUGGINSTALL where FLUGG installed,
should have subdir {bin|lib}/Linux-g++
if not given, 2-3 dir up + FLUGG from this script
[]
-o | --output=OUTPATH top of hierarchy where output will go
[/nova/data/condor-tmp/rhatcher/FLUGG]
-R | --rootsys=ROOTSYS where ROOT is installed
[/grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/v5-28-00]
--skindepth=0.77 use exponential horn skin depth of 0.77cm
[""] ""=use linear model
--fakeinfo=FAKEINFO configuration info for fake volume ["DEFAULT"]
use " XPOS YPOS ZPOS R DZ [MAXFAKEOUT] " units=cm
use "DEFAULT" to have it on, but use default
one word alternatives: "OFF" or "DISPLACED"
Dealing with Failures¶
Jobs fail for a number of reasons; some due to the batch system, some due exceptions in the FLUKA code. We only want to keep those files that ran to completion. Results from individual jobs come back as directories called RunXXXX collected under an umbrella directory. By using the read_the_whirlwind.sh script one can sort those results, weed out the bad files and make a directory of symlinks to all the root files, dk2nu files and/or target files.
cvs -d ${NUMI_BEAM_SIM_CVS} checkout -p -r proto2-numix g4numi_flugg/scripts/reap_the_whirlwind.sh > reap_the_whirlwind.sh
chmod +x reap_the_whirlwind.sh
bash reap_the_whirlwind.sh <directory-of-runs>
In the example below Run00002 failed in some identifiable way, and job 0001 was run without the option to produce the target file. Then the resulting directory structure might look like:
flugg_mn000z200i_rp11_lowth_pnut_f112c0f093bbird
Run0000
flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0000.{root|dk2nu.root|log|out|err}
target0000.root
<assorted-other-files>
Run0001
flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0001.{root|dk2nu.root|log|out|err}
<assorted-other-files>
Run0002.BAD
flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0002.{root|dk2nu.root|log|out|err}
<assorted-other-files>
Run0003
flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0003.{root|dk2nu.root|log|out|err}
target0003.root
<assorted-other-files>
...
dk2nu
symlink to ../Run0000/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0000.dk2nu.root
symlink to ../Run0001/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0001.dk2nu.root
symlink to ../Run0003/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0003.dk2nu.root
root
symlink to ../Run0000/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0000.root
symlink to ../Run0001/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0001.root
symlink to ../Run0003/flugg_mn000z200i_rp11_bs1.1_pnut_lowth_f112c0f093bbird_0003.root
target
symlink to ../Run0000/target0000.root
symlink to ../Run0003/target0003.root
Generally one only need supply the directory to be sorted, but there are some options. The --no-target option is useful if none of the jobs were run to generate target files, so it doesn't whinge so much.
reap_the_whirlwind.sh [options] <top-dir>
Purpose: Rummange around under <top-dir> for directories names Run????.
Determine if that particular flugg run failed (looking in err
logs); rename the director as Run????.Bad if so deemed.
Then make subdirectories "root" and "dk2nu" and populate
them with symlinks to remaining good files.
--revert restore all Run????.Bad directories to Run????
--no-bad don't try to classify runs (use w/ --revert to start fresh)
--no-flugg don't make flugg links
--no-dk2nu don't make dk2nu links
--no-target don't make target links
--no-refresh don't wipe flugg/dk2nu before starting
--maxlinks= limit # of symlinks (to equalize samples) [10000]
--agemin= minimum age of directory in minutes [10]
FLUKA FLUGG combinations known to work¶
| gcc 3.4.3 32-bit | flugg 2009_3 | flugg 2009_4 |
|---|---|---|
| fluka 2008.3d | known to run | |
| fluka 2011.2b.6 | runs | builds (MAGNEW log errors) |
A word about compilers and fluka/flugg¶
FLUGG currently (2014-08) can NOT be run 64-bit because it attempts to shove a pointer (which would be 64-bit) into a G4int (which is 32-bit).
Peanut¶
About the PEANUT option:
http://www.fluka.org/web_archive/earchive/new-fluka-discuss/2311.html note of possible interest
This is the line it adds:
if [ "$PEANUT" == "true" ]; then
echo "Special PEANUT run, model used for all energies."
echo "PHYSICS 1000. 1000. 1000. 1000. 1000. 1000. PEATHRES" >> ${inp}
fi