Project

General

Profile

History

Wiki » AnalysisWorkshop2017Feb »

Using Fermigrid to analyze your data

This section is intended to guide the user on how to use the provided scripts to submit jobs to Fermigrid.

Five types of jobs

In general, there are 5 types of jobs you may need to run:

  • MC truth generation
  • MC truth digitization & reconstruction
  • MC data analysis
  • DAQ data unpacking & reconstruction
  • DAQ data analysis

Since the nature of these jobs are different, the job submission script needs to be altered case by case.
In the future, a single script will be provided with options to specify which kind of job you want to submit.

Organization of the job submission script is shown in the figure below.

jobSubmissionScripts.png

Submit your first job for MCGen

To setup the plugins to use jobsub client,

source /grid/fermiapp/products/common/etc/setups.sh

To submit your first job, please go to the folder

gm2analyses/day1/talk5Fermigrid/MCGen

There are two scripts in that folder:

gm2analyses/day1/talk5Fermigrid/MCGen

You can submit gridjobs using the following command:

source gridSetupAndSubmit.sh mdc0 100 10

This will generate 10 gm2ringsim_mdc0.root files with 100 events each.

You can use the jobsub_q command to monitor your job

jobsub_q --user=${USER}

To fetch the log of a particular job,

jobsub_fetchlog -G gm2 --user=${USER} --jobid=${JOBID}

where ${JOBID} is something like 17919205.0@fifebatch2.fnal.gov.

Monitoring your job

1. General FIFEMON
https://fifemon.fnal.gov/monitor/dashboard/db/fifebatch

2. FIFEMON Job memory request and usage
https://fifemon-pp.fnal.gov/dashboard/db/job-memory-request-and-usage?var-Experiment=gm2

Using SAM datasets for your analysis

Below are the variables you need to set before you can use SAM. They are all set in the job submission script so you do not have to worry about them.

export SAM_EXPERIMENT=gm2
export EXPERIMENT=gm2
export IFDH_BASE_URI=http://samweb.fnal.gov:8480/sam/gm2/api
export JOBSUB_SERVER=https://fifebatch-dev.fnal.gov:8443
setup fife_utils

SAM datasets

I have created a dataset for MDC0 root files under the name MDC0ForWorkshop2017Feb.
So you can use it for your grid jobs.

Run over SAM datasets on pnfs

There are two scripts in the folder:

gm2analyses/day1/talk5Fermigrid/MCAna

Before submitting a job using local build/release, you need to install all your libraries, etc into the localProducts folder.

mrb i --generator ninja

You can submit grid jobs using the following command:

source gridSetupAndSubmitSAMDataset.sh ANALYZER_NAME SAM_DATASET_NAME

For instance, to analyze the SAM dataset MDC0ForWorkshop2017Feb using talk4CrystalHitAnalyzer,

source gridSetupAndSubmitSAMDataset.sh talk4CrystalAnalyzer MDC0ForWorkshop2017Feb

The gridSetupAndSubmitSAMDataset.sh will check the number of files in that SAM dataset and submit the same amount of jobs on the grid.

Handling job output files

Where to write your files (dCache/pnfs)

The Bluearc filesystem used to hold data for interactive jobs (/gm2/data) is too slow to handle the output from many parallel jobs. Trying to write your output will cause your jobs to sit and hang for a long time, meaning that they are taking up a CPU on the grid without really doing anything.

Instead, you should be writing the output of your files to dCache. dCache is not a filesystem in the traditional sense -- you cannot do everything you would do with a normal disk -- but it is fast enough to handle the output of many thousands of jobs simultaneously. The general rule-of-thumb is that the output of your grid jobs (it is setup in the gridSetupAndSubmit.sh) should be:

/pnfs/GM2/scratch/users/<username>

This area is large, but it is not persistent. As long as you are using your files they will stay there, but once you stop using them they are at risk of deletion (right now lifetimes are on the order of 1 month). You start by putting your files here so you can check that they are correct without using up space on tape. How to do that in the next section.

If you're writing your own job scripts, you will need to use ifdh cp to copy your output to dCache (see below for usage)

More details on the limitations of Bluearc and best practices can be found at the https://cdcvs.fnal.gov/redmine/projects/fife/wiki/FIFE_Data_Handling (FIFE data handling wiki).

Look at a single root file on dCache

If there is a root file you want to look at, you can use a the xrootd interface which allows root to stream files off of dCache. This can work even if you are not located at Fermilab. The utility pnfs2xrootd will convert a path in pnfs into a uri that xrootd can understand. You can then use this uri to open the file directly with root:

root `pnfs2xrootd /pnfs/GM2/scratch/users/<username>/myjob/job0.root`

Merge root files on dCache

If you want to combine the output of your jobs into a single root file on Bluearc, you can again do that using pnfs2xrootd:

hadd /gm2/data/users/<username>/combined.root `pnfs2xrootd /pnfs/GM2/scratch/users/<username>/myjob/*.root`

Note that the output must go to Bluearc, you cannot write directly to dCache using hadd.

Copy a limited number of files from dCache to Bluearc

If you really need to access the file directly, you can copy an individual file back to Bluearc (note, you should not copy large amounts of data back to Bluearc):

ifdh cp /pnfs/GM2/scratch/users/<username>/myjob/job0.txt /gm2/data/users/<user>/job0.txt

Note that the syntax of ifdh cp (documentation here) is not the same as regular cp, specifically it does not automatically recognize the destination as a directory. To copy a series of files into an existing directory, use:

ifdh cp -D file1 file 2 /some/directory/