Project

General

Profile

So You Want to Make Some Monte Carlo (A Quick Start Guide for Production)

Not running production jobs? Look here: So You Want to Make Some Monte Carlo (A Qwik Start Guide for non-Productioners)

So you need to make some Monte Carlo and perhaps don't have a month's time to devote learning all software properly. This guide is for you. The goal of this page is to get you started on generating some MC with minimal thought and effort.

There are two scripts you need to use to generate MC:
1. make_sim_fcl
- This script generates the fcls you need to process MC. This script resides in the "novaproduction" package. It is setup automatically to run without needing to locally install the package. You can verify this by using, which make_sim_fcl
2. submit_nova_art.py
- This script submits jobs to GRID for you based on an input configuration file that contains the SAM dataset definition generated from make_sim_fcl. Much like make_sim_fcl, this script works without you needing to setup the NovaGridUtils package.

Though you don't need to, I would still suggest setting up local copies of the both novaproduction and NovaGridUtils (addpkg_svn -h novaproduction) in a test release.

Are you making single-particle overlaid MC? Jump here: Single Particle Overlays

make_sim_fcl

Now let's go through some examples of how you'll be using the scripts above. The examples below were used in the Second Analysis Campaign and pulled directly from the ECL
When you're ready to run your make_sim_fcl command, you'll need a production proxy to be able to copy to the FTS dropbox on PNFS. Run

setup_fnal_security --production

first and you should be good to go.

Standard production make_sim_fcl settings for beam MC

Standard settings:

-fts --iteration 1 --batchnumber 1 --timespec full --generator genie

Real conditions, variable intensity:

# ND
--randomsubruns <nfiles> $NOVAPRODUCTION_DIR/goodruns/nd_grl_v0_epoch1-3c.txt --potperspill
# FD
--randomsubruns <nfiles> $NOVAPRODUCTION_DIR/goodruns/fd_grl_v1_epoch1-3c.txt

Events or POT per/file:

# ND
--pot 5e16 --potperspill
# FD
-n 1000

Detector:

#ND
-det nd
#FD
-det fd

Horn current:

-hp fhc 
# or
-hp rhc

Flux swap:

-fs nonswap
# or
-fs swap
# or ( swaps both nue and numu for tau)
-fs tau
# or ( swaps only numu for tau)
-fs numutau 

Overlay:

#ND rock FHC
--mix rock g4_secondaries_nd_genie_nonswap_fhc_R16-02-11-prod2genie.a_nova_v08_rocksingles_v1_rock_secondaries
# ND rock RHC
--mix rock g4_secondaries_nd_genie_nonswap_rhc_nova_v08_ideal_batch1_v1_ideal-gain100-rock_secondaries
# FD cosmic
--mix cosmic prod_artdaq_S15-03-11_fd_cosmic_epochs1-3c_v1_goodruns_snapshot20161128

Important note: the overlay definition you give needs to have an available snapshot. If you're not sure, run:

samweb take-snapshot <definition>

You will also want to make sure that this dataset has also been pre-staged from tape. You can check if it needs to be prestaged by listing a few filenames and checking them with:

cache_state.py -d <definition>

If the files are on tape you will need to follow the instructions for pre-staging the files in the SAM web cookbook.

examples using make_sim_fcl for ND

These are the three most requested samples for the ND

ND Genie non-swap no-RW (no-reweight) w/Rock Overlays:

make_sim_fcl -rsr 20000 $NOVAPRODUCTION_DIR/goodruns/nd_grl_v0_epoch1-3c.txt --pot 5e16 --potperspill -det nd -gen genie -hp fhc -fs nonswap -i 1 -fts -bn 3 -ts full -nogenierw --mix rock g4_secondaries_nd_genie_nonswap_fhc_R16-02-11-prod2genie.a_nova_v08_rocksingles_v1_rock_secondaries

ND Genie Fluxswap (no overlays):

make_sim_fcl -rsr 20000 $NOVAPRODUCTION_DIR/goodruns/nd_grl_v0_epoch1-3c.txt --pot 5e16 --potperspill -det nd -gen genie -hp fhc -fs swap -i 1 -fts -bn 1 -ts full -nogenierw

ND Genie nonswap w/RW (reweight) w/Rock Overlays:

make_sim_fcl -rsr 100000 $NOVAPRODUCTION_DIR/goodruns/nd_grl_v0_epoch1-3c.txt --pot 5e16 --potperspill -det nd -gen genie -hp fhc -fs nonswap -i 1 -fts -bn 4 -ts full --mix rock g4_secondaries_nd_genie_nonswap_fhc_R16-02-11-prod2genie.a_nova_v08_rocksingles_v1_rock_secondaries

ND CRY (gainmode 100):

The following command produces 5k files with 1M events a piece:
make_sim_fcl -gm 100 -lr 1000049 -ns 100 -n 1000000 -gen cry -fs all -det nd -i 1 -bn 1 -ts ideal -fts

examples using make_sim_fcl for FD

FD Genie nonswap (with cosmic overlay):

make_sim_fcl -rsr 6000 $NOVAPRODUCTION_DIR/goodruns/fd_grl_v3_020816.txt -n 1000 -det fd -gen genie -hp fhc -fs nonswap -i 1 -bn 1 -ts full -nogenierw -fts --mix cosmic prod_artdaq_S15-03-11_fd_cosmic_epochs1-3c_v1_goodruns_snapshot20161128

FD Genie fluxswap (with cosmic overlay):

make_sim_fcl -rsr 6000 $NOVAPRODUCTION_DIR/goodruns/fd_grl_v3_020816.txt -n 1000 -det fd -gen genie -hp fhc -fs swap -i 1 -bn 1 -ts full -nogenierw -fts --mix cosmic prod_artdaq_S15-03-11_fd_cosmic_epochs1-3c_v1_goodruns_snapshot20161128

FD Genie tauswap (with cosmic overlay):

make_sim_fcl -rsr 6000 $NOVAPRODUCTION_DIR/goodruns/fd_grl_v3_020816.txt -n 1000 -det fd -gen genie -hp fhc -fs tau -i 1 -bn 1 -ts full -nogenierw -fts  --mix cosmic prod_artdaq_S15-03-11_fd_cosmic_epochs1-3c_v1_goodruns_snapshot20161128

FD CRY gainmode 100:

make_sim_fcl -gm 100 -lr 1000049 -ns 100 -n 200 -gen cry -fs all -det fd -i 1 -bn 1 -ts ideal -fts

FD CRY gainmode 140:

make_sim_fcl -gm 140 -lr 2000049 -ns 100 -n 200 -gen cry -fs all -det fd -i 1 -bn 1 -ts ideal -fts

Custom modifications to the fhicl file

make_sim_fcl uses a template to make fhicl files. By default the template lives in $NOVAPRODUCTION_DIR/utils/sim_fcl.template. If you need to customize the fhicl for your job, you should make a local copy of this template, and then tell the job to use it with:

make_sim_fcl --templatefhicl some_custom_fhcl.template ...

submit_nova_art.py

I suggest setting up a local copy of NovaGridUtils for this if you haven't already. In the configs directory of NovaGridUtils there are two configuration templates ready to use, prod_mcgen_template.cfg and prod_nd_genie_overlay_template.cfg. If you don't want download the package, you can just copy them from the latest NGU package at $NOVAGRIDUTILS_DIR/configs/

The syntax for running jobs is, submit_nova_art.py -f /<path_to>/<config_file>.cfg

Standard configurations (non-overlays)

I refer to non-overlay MC jobs as standard jobs, this is typically anything made for FD as well as certain ND samples (Fluxswap, CRY)
For these standard jobs, you'll want to use $NOVAGRIDUTILS_DIR/configs/prod_mcgen_template.cfg. This template has instructions on how to fill it out that are hopefully very straightforward. Much of the boiler plate has now been moved into includes so only the options you need to change are seen in the template.

#This is a template to help you configure you mc generation jobs on the grid.

# Include standard MC generation configuration
-c mcgen

########################################
# General options that you need to set #
########################################

#Type the name of your job here
--jobname <job_name>

#Type the definition that contains the fcl you want to process
--defname <dataset_definition_name>

#This is where you place which tag you want to generate in
--tag <novasoft_release>

#Number of jobs you'd like to submit
--njobs <num_jobs>

####################################
# Default options you might change #
####################################

# By default, run MC generation off-site
-f everywhere.inc
#-f onsite.inc OR offsite.inc

# Official production job?
-f production.inc

# Normally produce artdaq but...
--outTier=out1:artdaq

# ...switch to g4 for making rock MC and other overlay pieces
#--outTier=out1:g4

Overlay configurations

Overlays are more complex jobs to run than standard configurations. Once again the template below should help get you started. Again, much of the boiler plate has now been moved into includes so only the options you need to change are seen in the template.

Important note:

#This is a template to help you configure you mc generation jobs on the grid.

# Include standard MC generation configuration
-c mcgen

########################################
# General options that you need to set #
########################################

#Type the name of your job here
--jobname <job_name>

#Type the definition that contains the fcl you want to process
--defname <dataset_definition_name>

#This is where you place which tag you want to generate in
--tag <novasoft_release>

#Number of jobs you'd like to submit
--njobs <num_jobs>

####################################
# Default options you might change #
####################################

# By default, run MC generation off-site
-f everywhere.inc
#-f onsite.inc OR offsite.inc

# Official production job?
-f production.inc

#########################
# Overlay configuration #
#########################

# Choose your overlay procedure. 
# The script(s) you choose here should
# match the settings used when making
# your fhicl files

#--mix rock_overlay.sh
#--mix cosmic_overlay.sh
#--mix rock_overlay.sh:cosmic_overlay.sh
#--mix singles_overlay.sh

# Overlay jobs always produce artdaq files and can
# only be run one at a time. Always set these for
# overlay jobs.
--outTier=out1:artdaq
--files_per_job 1

If you need to run with a non-standard overlay rate, you should copy the contents of $NOVAGRIDUTILS_DIR/configs/base/fhc_overlay.inc into your configuration file and modify as needed. This should also include the base overlay boilerplate via -f rock_overlay.cfg which you are very unlikely to need to modify.

Some broad strokes on how overlay works.

We make fcl definitions using `make_sim_fcl`. When doing this we pass it a cosmic definition to use. This is because when we generate Monte Carlo, we simulate GENIE events in the FD and then we overlay cosmic data on top of that. Overlay means that we literally place a cosmic event on top of the GENIE MC event we have simulated.

The fcl files then contain the actual cosmic file which should be loaded in, which is contained in the cosmic definition used above.

Obviously we have tons and tons of cosmic files, however we only need a small subset for the overlay process. We therefore need to figure out which files we need to pull off tape. This is done using `overlay_prestage_def.py`

We then need to pull those files off tape. This is done using `prestage-definition -d <defname> --parallel 5`

Then when we come to actually run the generation jobs, we use the fcl definition which was produced by `make_sim_fcl` as the input.
In each of these fcls files it will tell the grid job which cosmic file to read in so that the overlay process can occur.

— For the ND we just skip every part above that has overlay in it. But the generic process is the same.

Final Notes

You should now have everything you need to run jobs. The output of submit_nova_art.py should have given you a jobsubID and a samweb monitoring page. I suggest saving them somewhere so you can access them at a later date if needed. Additionally, don't forget to fill out an ELOG entry in the ECL, so that other may learn from your work and Conveners can track your progress.

If you need to make something different from the above, such as a systematics sample or some other special sample, you'll probably using the --syst option.

For a list of stand/basic options plus more examples, use:
make_sim_fcl -h
For a full list of options and brief explanations, use:
make_sim_fcl -ah

If you know you will be running lots of MC, it would behoove you to learn more about how make_sim_fcl works: Running make_sim_fcl as well as just setup up the head version (described near the top) and just reading the code.