Project

General

Profile

CAFAna on the grid » History » Version 14

Version 13 (Christopher Backhouse, 05/22/2020 05:25 PM) → Version 14/17 (Miriama Rajaoalisoa, 09/10/2020 03:04 AM)

h1. CAFAna on the grid

First, are you sure you need to run on the grid?

How long is your macro expected to take (look at the progress bar)? If it's less than about an hour the grid is definitely more hassle than it's worth (not for CAFAna reasons, just in general).

Second, are you _really_ sure you need to run on the grid?

The goal of CAFAna is for macros to run relatively quickly interactively. If you have very large runtimes there may well be something you can do.

h2. Optimized build

Probably doesn't help too much if your problem is large file sizes, but it can be a big help (2-3x) in other cases.

When you setupnova

<pre>
setupnova -r $REL -b maxopt
</pre>

Or you can do it later
<pre>
export SRT_QUAL=maxopt
srt_setup
</pre>

Remember if you're using a test-release your local changes won't have any effect unless you've built in the same version (debug vs maxopt) as you're running.

h2. Running over fewer files

The least satisfying solution, but also the easiest. Can you get away with running over fewer files? Pass an option like @-s 10@ to @cafe@ to run over only 1/10th of the files in your dataset or wildcard.

Particularly when developing your macro you want as quick a turnaround as possible, and probably don't care so much what the results actually look like.

If you spend most of your time rerunning a multiple-minutes macro to make a series of changes to the plotting styles it will be well worth your time to split your macro up. A good pattern is one (slower) macro that simply does all the loading from files and immediately writes out all the @Spectrum@ or @Prediction@ etc objects you need using @SaveTo@, and a second (very quick) macro to load those in (@LoadFrom@) and plot them, that you can iterate as many times as necessary.

h2. Reducing file size (sumdecaf/concat)

Take a look at your input files

<pre>
samweb list-definition-files --summary my_query
</pre>

or

<pre>
du -sh my_dir/
</pre>

If your dataset is large (more than a gigabyte or so) you can definitely go faster with smaller files.

Are there sumdecafs/concats you should be using rather than CAFs? The analysis groups take the CAFs produced by production and produce analysis specific sumdecafs/concats where a preselection has been applied (some combination of quality, containment and loose cosmic rejection cuts). These files also drop some excessive truth information carried around by full CAFs. They are usually substantially smaller than the corresponding CAFs.

If the existing sumdecafs are unsuitable for you for some reason, _e.g._ they cut out events you need, or you could make your files substantially smaller by applying more stringent cuts, look into making your own sumdecafs ([[Making decafs]]). It's not that hard, and could save you a lot of time in future. If your sumdecafs might be broadly useful, please discuss the cuts and metadata with the relevant people (group conveners, production conveners) and look into making them broadly available.

h2. Reducing file number (concat)

If you have a large number of input files (thousands) time can be dominated by opening and closing files, especially from pnfs over xrootd, instead of doing useful work. If it only takes a second or so to process each individual file concatenating your inputs could make a big difference. Production should provide a concatenated version of every completed decaf dataset for the Second Analysis onwards at @/pnfs/nova/production/concat/@ and @/nova/prod/concat/@. If you have some unusual requirements, or you need to concatenate your own decafs, look at [[Concatenating CAFs]].

h2. Reducing release tarball size

Since the jobs can no longer access the bluearc areas directly, when submitting jobs, your local release will be tarred up and sent to the worker nodes. Occasionally, however, your test release can be bloated by temporary files or other files that are not necessary for running on the grid. Creating a tarball of such sizeable release can fill up the temporary directory on the GPVM nodes and, hence, prevent you from submitting jobs. The script "testrel_tarbal":https://cdcvs.fnal.gov/redmine/projects/novaart/repository/entry/trunk/NovaGridUtils/bin/testrel_tarball is useful in this case, as it allows you to reduce your release tarball size. It trims down the release directory by ignoring common files and directories not required for grid running, these includes any temporary directories, documentation or manual. Besides, intermediate build files (*.o and *.d) and debug build (*debug) will not be included in the produced tarball. "testrel_tarbal":https://cdcvs.fnal.gov/redmine/projects/novaart/repository/entry/trunk/NovaGridUtils/bin/testrel_tarball also minimizes libraries and executable' sizes via the GNU utility @strip@ command. To run the script, you need to specify your test release directory and the directory at which to place the output tarball :

<pre>
testrel_tarball </path/to/test_release> </path/to/output_tarball>
</pre>

This will produce a tarball @output_tarball.tar.bz2@ at the specified output directory. Then you can specify to @submit_cafana.py@ to use this user-made tarball with the option :
<pre>
--user_tarball </path/to/output_tarball>
</pre>

It should be noted that the @--user_tarball@ option conflicts @--testrel@. Therefore, once ou specify a particular tarball to use, you do not need to specify the test release directory @-t DIR@ or @--testrel DIR@ anymore. In the same token, the @--reuse_tarball@ is also redundant with @--user_tarball@.

In the case of a failed job submission, it is always good to check the temporary directory on the GPVM nodes with :
<pre>
ls -lhS /tmp
</pre>
and make sure to remove any abnormally large resulting temporary files.

h1. So you really do need to run on the grid

First, all the regular grid rules apply "(LEARN MORE)":https://cdcvs.fnal.gov/redmine/projects/novaart/wiki/Using_Condor_and_Running_on_Grid_from_IF_Cluster_Machines. Remember that the bluearc areas are completely inaccessible for reading/writing/execution, though you should *never* read any large files from bluearc anyway. Best practice is to write your output files to @/pnfs/nova/scratch/users/$USER/@. If you need to be certain that you keep the files around for a while, either transfer them to @/pnfs/nova/persistent/users/$USER/@ or @/nova/ana/users/$USER/@ but remember to tidy these files up as the disks are limited.

@submit_cafana.py@ is much like @submit_nova_art.py@. It will submit N copies of your macro to the grid. Each one will automatically set the @--stride@ and @--offset@ arguments to @cafe@ so that between them the jobs see all the files in whatever datasets the macro uses, but divided evenly between them. @submit_cafana.py@ automatically copies all output files (@*.root@ but also _e.g._ @*.txt@ in case you're making event lists) to the specified output directory. Outputs will be named _e.g._ @myoutput.1_of_10.root@ automatically. It also copies back a log file for each process.

<pre>
usage: submit_cafana.py [-h] [-f FILE] -n N -r REL [-i INPUT_FILE] -o DIR
[-t DIR] [-ep PRODUCT:VERSION] [-ss] [-off] [-d] [-x]
[--dedicated] [--disk DISK] [--memory MEMORY]
[--lifetime LIFETIME] [--source SOURCE]
[--reuse_tarball | --user_tarball USER_TARBALL]
[--print_jobsub] [--test] [--ifdh_debug]
macro.C [args [args ...]]

Submit a CAFAna macro with datasets split between N jobs

positional arguments:
macro.C The CAFAna macro to run
args Arguments to the macro

optional arguments:
-h, --help show this help message and exit
-f FILE, --file FILE Text file containing any arguments to this utility
-n N, --njobs N Number of grid processes
-r REL, --rel REL Release to use
-i INPUT_FILE, --input_file INPUT_FILE
Copy this input file to work area on worker node
-o DIR, --outdir DIR Directory output files will go to
-ss, --snapshot Use latest snapshot instead of requerying
-off, --offsite Run this cafana job offsite
-d, --drain Recover files missing from your output directory
-x, --xrootdebug Add extra xrootd debuggin information

Job control options:
These optional arguments help control where and how your jobs land.

-t DIR, --testrel DIR
Use a test release at location TESTREL. It will be
tarred up, and sent to the worker node. Conflicts with
--user_tarball
--user_tarball USER_TARBALL
Use existing test release tarball in specified
location rather than having jobsub make one for you
(conflicts with --testrel, and is redunant with
--reuse_tarball)
--reuse_tarball Do you want to reuse a tarball that is already in
resilient space? If using this option avoid trailing
slash in --testrel option. (redundant with
--user_tarball)
--dedicated Only run on dedicated nodes on fermigrid (default is
to run opportunistically)
--disk DISK Local disk space requirement for worker node in MB
(default is 2000MB).
--memory MEMORY Local memory requirement for worker node in MB
(default is 1900MB).
--lifetime LIFETIME Expected job lifetime. Valid values are an integer
number of seconds. (default is 10800=3h)
--source SOURCE Source script SOURCE:par1:par2:..
-ep PRODUCT:VERSION, --extproduct PRODUCT:VERSION
Setup this external product on the worker node in
format <product>:<version>

Debugging options:
These optional arguments can help debug your submission.

--print_jobsub Print jobsub command
--test Do not actually do anything, just run tests and print
jobsub cmd
--ifdh_debug Verbose output for pinning down IFDH/dCache issues
</pre>

To prevent multiple simultaneous SAM request hammering the SAM database, CAFAna automatically detects when it is being run on the grid and shares a single snapshot of each dataset among all the processes.

Once the jobs are complete you can combine the results using @hadd_cafana@. This behaves exactly the same as regular @hadd@, except it can handle the @TObjString@ contents in CAFAna output files that cause regular @hadd@ problems. In particular, all it does is sum the various histograms, so you can only combine things where that does something sensible. This is true for regular @Spectrum@ objects, and for @PredictionNoExtrap@. It's not true right now for _e.g._ @MichelDecomp@, but in general these problems can be resolved by changes in the serialization format of unsuitable classes.



h2. Tips

# Make sure to run your @hadd_cafana@ command on a BlueArc disk (preferably @/nova/ana/users/$USER/...@). Avoid using dCache (@/pnfs/nova/...@).
# If combining 1000 files, it is more efficient to run 10 incantations of @hadd_cafana@ over a reasonable number of files (O100), and then combining them, versus running a single command over all 1000 files. This is because of a restriction on the number of simultaneous disk access requests allowed on dCache. Coding a simple bash/python script to do this shouldn't be too difficult.
# You can @hadd_cafana@ over files in pnfs with
<pre>
hadd_cafana out.root `pnfs2xrootd /pnfs/.../in*.root`
</pre>
# To view your log files, @cat@ doesn't work from pnfs, but you can @dccp@ the file to stdout, which has the same effect
<pre>
dccp /pnfs/.../log.txt -
</pre>
# To submit a macro containing a custom header file you need to move both the macro and header file to your scratch area. All files you send to the grid are put in $CONDOR_DIR_INPUT so for your include statement in your macro you don't need the full file path just #include "header.h" is all you need
<pre>
submit_cafana.py -n <jobs> -r <rel> -i </path/to/header/in/scratch>/header.h -o <output directory> macro.C
</pre>
# Similarly, if you need to include an input file into the macro, you will need to move that file and your macro to your scratch area and include that input file using the -i option in the same way as above. Additionally, inside your macro where you open up the input file, you will need to open it via
<pre>
TFile *f1 = TFile::Open(pnfs2xrootd("/pnfs/nova/scratch/path/to/file/input.root").c_str());
</pre>