Generating MCC8 Detector Systematic and Reco Variation Samples¶
These instructions are for the generation of systematic variation (g4, det sim, and reco1). There are some general comments about grid submission snuck into the instructions, so enjoy and learn!
Configuring your project.py xml file¶
There is a template file to base you project xml file off of located here:
Copy these files into your app volume area:
mkdir -p /uboone/app/users/$USER/mcc8.0_variation_samples/ cd /uboone/app/users/$USER/mcc8.0_variation_samples/ cp /uboone/app/users/uboonepro/mcc8.0_variation_samples/* \ /uboone/app/users/$USER/mcc8.0_variation_samples/
Note that the /uboone/app/users/ area should be used to store configuration files, uboonecode builds, but should never be used to store data files. Everywhere in the xml file that you find CHANGE you should determine if you need to adjust the parameter below it:
<!-- CHANGE YOUR USER NAME --> <!ENTITY user_name "users/joelam">
In this example, you should change "joelam" to your user name. You can find a list of the fcl files for each variation here:
Once you have edited your file, then you need to make sure that the staging and output directories are ready. You credit those files with these commands:
mkdir -p /pnfs/uboone/scratch/users/$USER #This creates a directory in the dCache scratch area mkdir -p /uboone/data/users/$USER #this creates a directory in the BlueArc data volume
Note that the dCache scratch area does not have a quota, but files have a limited lifetime before they are flushed from the volume (usually ~ 30 days). But there is NO guarantee that files will be stored permanently. While on the BlueArc data volume, there is a 1.5 TiB quota for each user but the storage is permanent. For full details look here: https://redmine.fnal.gov/redmine/projects/fife/wiki/Understanding_storage_volumes
Now you're ready to submit jobs. First setup the environment with a current version of uboonecode so that you get a current version of the larbatch UPS product.
source /cvmfs/uboone.opensciencegrid.org/products/setup_uboone.sh setup uboonecode v06_26_01_02 -q e10:prof cd /uboone/app/users/$USER/mcc8.0_variation_samples/ projectgui.py <insert_xml_file_name_here>
Once the window comes up, you will need to click on a stage: sim, reco, or mergeana. Once you click on a stage (you'll start with sim), then you can click on submit button in the top right. You should see 120 jobs in the some of Idle, Running, and Held. Hopefully, none go held, but we'll deal with that later.
You can monitor the status of a project in the gui by clicking on "Update", and once there are no longer any idle, running, or held jobs, then you should run "Check". If there are no errors, then you are ready to create a dataset from the output. Go to the directory specified in <outdir> in the stage your working on.
#### <outdir>/pnfs/uboone/scratch/&user_name;/&tag;/&relsim;/sim/&name;</outdir> cd /pnfs/uboone/scratch/users/kirby/mcc8_syst/v06_26_01_02/sim/prodcosmics_corsika_cmc_uboone_syst/
You should see a bunch of .root output files. And we need to move them to permanent storage. We will do this using SAM4Users utilities. http://microboone-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=6896 These commands will have to be translated slightly but hopefully you understand what is being done. First we will make text files that contain the full paths to the files we've generated:
###ls /pnfs/uboone/scratch/users/&user_id;/&tag;/&relsim/sim/&name;/*/&name;*.root \ ### >& /uboone/app/users/$USER/&name;_sim_filelist.txt #this has to be translated and using the example above would become ls /pnfs/uboone/scratch/users/kirby/mcc8_syst/v06_26_01_02/sim/prodcosmics_corsika_cmc_uboone_syst/*/prod*.root \ >& /uboone/app/users/$USER/mcc8.0_variation_samples/prodcosmics_corsika_cmc_uboone_syst_sim_filelist.txt
Check to make sure that the files in that *filelist.txt file are the correct output artroot files and not small histogram files. You may need to change the "ls" command to fix that. Next we'll setup fife_utils and declare the files to SAM4Users.
setup fife_utils sam_add_dataset -n prodcosmics_corsika_cmc_uboone_mcc8_syst_sim -f \ /uboone/app/users/$USER/mcc8.0_variation_samples/prodcosmics_corsika_cmc_uboone_syst_sim_filelist.txt sam_move2archive_dataset -n prodcosmics_corsika_cmc_uboone_mcc8_syst_sim
At that point, the files are removed from dCache scratch space and moved to tape-backed, permanent storage. You will now need to access them through the SAM datasets definitions. To do this, you should add the following lines in the reco stage of the xml project file:
<inputdef>prodcosmics_corsika_cmc_uboone_mcc8_syst_sim</inputdef> <jobsub_start>--expected-lifetime=short</jobsub_start> <maxfilesperjob>1</maxfilesperjob>