Using LBNE dCache Scratch Space at Fermilab¶

dCache Space¶

For temporary storage, you can put files in

     /pnfs/lbne/scratch/users/<makeyourowndirectory>

As of June 2015, the lifetime of files in the scratch area is approximately one month. An automatic process will clean up old files.

For persistent storage, you can put files in

     /pnfs/lbne/persistent/users/<makeyourowndirectory>

Warning about mv'ing files from scratch to persistent areas -- as the file is not rewritten, it retains its deletion policy. A file may then get removed from the persistent area if it started out in scratch and mv'd there. If you need to transfer a file from scratch to persisten, ifdh cp it and delete the scratch version.

Accessing dCache Space¶

With SLF6, which uses NFSV4, you can access files interactively in dCache directly using regular POSIX access -- ls, cp, rm, etc. all should work normally. But see this presentation for best practices and a description of what's under the hood, as to why some usage patters will be more efficient than others: https://indico.fnal.gov/getFile.py/access?contribId=30&resId=0&materialId=slides&confId=9737

For batch access, please use ifdh cp to copy files to and from dCache. This is also true of /lbne/data and /lbne/data2, whose NFS mounts are being removed from the local grid nodes (and are not available offsite anyway). IFDH requires a grid proxy (see below), but you should have this all set up by jobsub in a running job.

Older Instructions: Copying files to and from dCache¶

You will need to be a member of the LBNE Virtual Organization. Follow the
instructions for getting grid permissions to become a member:
Grid Setup Instructions

Instructions on how to copy from and write to dCache.  The exmaple
below was run February 2014 on lbnegpvm02, and the username is qzli
in order to refer to specificfiles on BlueArc and the dCache user scratch area.

As of Febraury 2014, the /pnfs/lbne/scratch/users is 6 PB big and is shared
by all Intensity Frontier experiments.  It is not backed up by tape -- please
use SAM for that.  Once SL6 is available, we can use NFS directly instead of
ifdh to access this disk.

  How to copy files to and from dCache.  Files can be directly transferred to dCache
from batch jobs without going through BlueArc first.

 *** SET THE WORKING ENVIRONMENT ***

> export ROLE=Analysis
> export EXPERIMENT=lbne
> kx509
> voms-proxy-init -rfc -noregen -voms lbne:/lbne/Role=$ROLE -valid 24:00
> voms-proxy-info -all

> . /grid/fermiapp/products/common/etc/setups.sh
> setup ifdhc
> which ifdh
 (make sure your ifdhc is from v1_3_1 or higher)

*** COPY FROM dCache ***

> TESTFILE=/pnfs/lbne/mc/lbne/simulated/001/lbne_test_file.root
> ifdh cp "${TESTFILE}" "/lbne/data/users/qzli/TESTFILE.root" 
> ls -l /lbne/data/users/qzli/TESTFILE.root

 *** COPY TO dCache ***

> ifdh mkdir /pnfs/lbne/scratch/users/qzli
> ifdh cp /lbne/data/users/qzli/TESTFILE.root /pnfs/lbne/scratch/users/qzli/
TESTFILE.root
> ls -l /pnfs/lbne/scratch/users/qzli/TESTFILE.root

Using xrootd to analyze files in dCache without copying them locally¶

Here's an example browsing a rootfile in dCache. Note the different pnfs directory name.

> kx509
> voms-proxy-init -rfc -noregen -voms lbne:/lbne/Role=Analysis
> root root://fndca1.fnal.gov:1094/pnfs/fnal.gov/usr/lbne/scratch/users/qzli/TESTFILE.root

Links to documentation:¶

http://trac.dcache.org/wiki/xrootd

https://srm.fnal.gov/twiki/bin/view/DcacheCorner/DcacheFAQ

See also LBNE DocDB 9657

dCache files can be accessed via posix interfaces on SL6, running NFSV4