Using Fermigrid to analyze your data¶
This section is intended to guide the user on how to use the provided scripts to submit jobs to Fermigrid.
Five types of jobs¶
In general, there are 5 types of jobs you may need to run:
- MC truth generation
- MC truth digitization & reconstruction
- MC data analysis
- DAQ data unpacking & reconstruction
- DAQ data analysis
Since the nature of these jobs are different, the job submission script needs to be altered case by case.
In the future, a single script will be provided with options to specify which kind of job you want to submit.
Organization of the job submission script is shown in the figure below.
Submit your first job for MCGen¶
To setup the plugins to use jobsub client,
source /grid/fermiapp/products/common/etc/setups.sh
To submit your first job, please go to the folder
gm2analyses/day1/talk5Fermigrid/MCGen
There are two scripts in that folder:
gm2analyses/day1/talk5Fermigrid/MCGen
You can submit gridjobs using the following command:
source gridSetupAndSubmit.sh mdc0 100 10
This will generate 10 gm2ringsim_mdc0.root
files with 100 events each.
You can use the jobsub_q
command to monitor your job
jobsub_q --user=${USER}
To fetch the log of a particular job,
jobsub_fetchlog -G gm2 --user=${USER} --jobid=${JOBID}
where ${JOBID}
is something like 17919205.0@fifebatch2.fnal.gov
.
Monitoring your job¶
1. General FIFEMON
https://fifemon.fnal.gov/monitor/dashboard/db/fifebatch
2. FIFEMON Job memory request and usage
https://fifemon-pp.fnal.gov/dashboard/db/job-memory-request-and-usage?var-Experiment=gm2
Using SAM datasets for your analysis¶
Below are the variables you need to set before you can use SAM. They are all set in the job submission script so you do not have to worry about them.
export SAM_EXPERIMENT=gm2
export EXPERIMENT=gm2
export IFDH_BASE_URI=http://samweb.fnal.gov:8480/sam/gm2/api
export JOBSUB_SERVER=https://fifebatch-dev.fnal.gov:8443
setup fife_utils
SAM datasets¶
I have created a dataset for MDC0 root files under the name MDC0ForWorkshop2017Feb
.
So you can use it for your grid jobs.
Run over SAM datasets on pnfs¶
There are two scripts in the folder:
gm2analyses/day1/talk5Fermigrid/MCAna
Before submitting a job using local build/release, you need to install all your libraries, etc into the localProducts folder.
mrb i --generator ninja
You can submit grid jobs using the following command:
source gridSetupAndSubmitSAMDataset.sh ANALYZER_NAME SAM_DATASET_NAME
For instance, to analyze the SAM dataset MDC0ForWorkshop2017Feb
using talk4CrystalHitAnalyzer,
source gridSetupAndSubmitSAMDataset.sh talk4CrystalAnalyzer MDC0ForWorkshop2017Feb
The gridSetupAndSubmitSAMDataset.sh
will check the number of files in that SAM dataset and submit the same amount of jobs on the grid.
Handling job output files¶
Where to write your files (dCache/pnfs)¶
The Bluearc filesystem used to hold data for interactive jobs (/gm2/data) is too slow to handle the output from many parallel jobs. Trying to write your output will cause your jobs to sit and hang for a long time, meaning that they are taking up a CPU on the grid without really doing anything.
Instead, you should be writing the output of your files to dCache. dCache is not a filesystem in the traditional sense -- you cannot do everything you would do with a normal disk -- but it is fast enough to handle the output of many thousands of jobs simultaneously. The general rule-of-thumb is that the output of your grid jobs (it is setup in the gridSetupAndSubmit.sh) should be:
/pnfs/GM2/scratch/users/<username>
This area is large, but it is not persistent. As long as you are using your files they will stay there, but once you stop using them they are at risk of deletion (right now lifetimes are on the order of 1 month). You start by putting your files here so you can check that they are correct without using up space on tape. How to do that in the next section.
If you're writing your own job scripts, you will need to use ifdh cp to copy your output to dCache (see below for usage)
More details on the limitations of Bluearc and best practices can be found at the https://cdcvs.fnal.gov/redmine/projects/fife/wiki/FIFE_Data_Handling (FIFE data handling wiki).
Look at a single root file on dCache¶
If there is a root file you want to look at, you can use a the xrootd interface which allows root to stream files off of dCache. This can work even if you are not located at Fermilab. The utility pnfs2xrootd will convert a path in pnfs into a uri that xrootd can understand. You can then use this uri to open the file directly with root:
root `pnfs2xrootd /pnfs/GM2/scratch/users/<username>/myjob/job0.root`
Merge root files on dCache¶
If you want to combine the output of your jobs into a single root file on Bluearc, you can again do that using pnfs2xrootd:
hadd /gm2/data/users/<username>/combined.root `pnfs2xrootd /pnfs/GM2/scratch/users/<username>/myjob/*.root`
Note that the output must go to Bluearc, you cannot write directly to dCache using hadd.
Copy a limited number of files from dCache to Bluearc¶
If you really need to access the file directly, you can copy an individual file back to Bluearc (note, you should not copy large amounts of data back to Bluearc):
ifdh cp /pnfs/GM2/scratch/users/<username>/myjob/job0.txt /gm2/data/users/<user>/job0.txt
Note that the syntax of ifdh cp (documentation here) is not the same as regular cp, specifically it does not automatically recognize the destination as a directory. To copy a series of files into an existing directory, use:
ifdh cp -D file1 file 2 /some/directory/