Project

General

Profile

Removing Bluearc Dependencies

As of January 18, 2018 it will no longer be possible to access BlueArc disk in any way from within a grid job, even through ifdh cp. This means that workflows doing things like (these are just illustrative examples; if you don't see exactly what you're doing here, don't simply assume you are safe)

source /grid/fermiapp/products/larsoft/setup

or

/experiment/app/users/username/foo/custom_code input_file ...

or even

ifdh cp -D /experiment/app/users/${GRID_USER}/foo/some.file.tgz ./
tar xzf some.file.tgz
./something_from_the_tar_file ...

will no longer work after the January downtime. Any reference to a /grid/data/foo, /grid/fermiapp/foo, or /experiment/app/foo location needs to be removed from job scripts. In nearly all cases, there are already drop-in replacements available. Broadly speaking we can consider three types of accesses that need to change: 1) Initial UPS setup calls (from common areas or experiment-specific areas), 2) Copies from Bluearc locations of tarballs containing custom code, executables, or other input files, and 3) scripts that currently directly read files from a Bluearc location (see the second example above.)

Replacing UPS or experiment software setup scripts

Please consult the following table to see how to modify your script to eliminate Bluearc-based setup calls in favor of CVMFS-based setups, which will work both on-site and off-site. These are just examples; contact your CS liaison or offline coordinator to find out what your experiment-specific areas are.

If you are doing... Replace it with...
source /grid/fermiapp/products/common/etc/setup(s) source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setup(s)
source /grid/fermiapp/products/larsoft/setup source /cvmfs/fermilab.opensciencegrid.org/products/larsoft/setup
source /grid/fermiapp/products/experiment/some_setup_script /cvmfs/experiment.opensciencegrid.org/foo/some_setup_script
(check with CS liaison or offline coordinator for full paths)

For source /grid/data/experiment/foo scripts, in general look for a CVMFS path within your experiment's area, but talk with your liaison of offline coordinator to find out what your proper path is. If it's still unclear, contact FIFE support.

Replacing tarball copies

If you are doing something like the following in your script:

ifdh cp -D /experiment/app/users/${GRID_USER}/foo/some.file.tgz ./
tar xzf some.file.tgz
#Do something with stuff from the tar file ...

You simply need to change the tar file source location to be something in a dCache (/pnfs/experiment) area. Typically one would use scratch or resilient dCache for this type of access, but you should consult your experiment's CS liaison of offline coordinator to confirm your experiment's policy. In that case your code should now look something like this (replace the /pnfs/ path as appropriate for your experiment:

ifdh cp -D /pnfs/experiment/resilient/users/${GRID_USER}/foo/some.file.tgz ./
tar xzf some.file.tgz
#Do something with stuff from the tar file ...

Replacing direct reads

Oftentimes one may have a custom or test release on an /experiment/app or /grid/fermiapp area. As an example, image one has a test release in

 /experiment/app/users/username/foo/local_build 
. In a grid job script one might do something like
Finally, let's consider a toy workflow that does something like this:

export LD_LIBRARY_PATH=/experiment/app/users/username/foo/local_build/lib:${LD_LIBRARY_PATH}
export PATH=/experiment/app/users/username/foo/local_build/bin:${PATH}
custom_code input_file ...

Where custom_code is an executable in the /experiment/app/users/username/foo/local_build/bin directory. The custom_code executable is accessed directly, as opposed to being copied to the worker node inside of a tar file. To make this work in a world without BlueArc, you can create a tarball of this area, put it in dCache, and copy the tar file into your local directory. After extracting the tarball you can simply treat the resulting area like your own work area before. Here is an example of what you can do:

$ cd /experiment/app/users/username/foo/local_build
$ tar --exclude '.git' --exclude '.svn-base' czf mytar.tar.gz bin lib other_files_as_needed

The lib, lib, and other_files_as_needed are just examples; you will need to modify this depending on your work area. We recommend excluding any .git directories as you won't need them when actually running the job, and in some packages they can actually be large relative to the rest of the area. Since these tar files will go out to every job, it's important to keep them as small as possible and only include things that are needed.

Once you have mytar.tar.gz, copy it to a dCache location with ifdh cp. Then, you can copy it into the job and set environment variables appropriately either with an ifdh command in your script:

ifdh cp -D /path/in/dcache/mytar.tar.gz .
tar zxf mytar.tar.gz
export PATH=${PWD}/bin:${PATH}
export LD_LIBRARY_PATH=${PWD}/lib:$LD_LIBRARY_PATH
run whatever else you normally run...

Or you can have jobsub do the initial copy for you:

 jobsub_submit <usual options> -f /path/in/dcache/mytar.tar.gz <rest of usual command> 

Then you only need to extract the tarball and set variables as appropriate:

tar zxf ${_CONDOR_DIR_INPUT}/mytar.tar.gz . # extracts the tarball into the top level directory 
export PATH=${PWD}/bin:${PATH}
export LD_LIBRARY_PATH=${PWD}/lib:$LD_LIBRARY_PATH
run whatever else you normally run...

Other notes

1) An additional note about environment variables: the PATH and LD_LIBRARY_PATH variables may not be enough to make your code work correctly. Other might include PYTHONPATH, for example.
2) If you have a local setup script that you run to set up your area, you could include that in the tar file and run it after extraction. It should work fine as long as it uses relative paths instead of hard-coded full paths.
3) The above examples are for illustrative purposes only. Your code might look somewhat different. Do not assume that your workflow is OK just because it doesn't look exactly like this. As a rule, if anything references a location in /experiment/app, /grid/fermiapp, or /grid/data, it is going to have to change. Another possibility is that your script could be calling other scripts, and those other scripts will also need to change. Contact your CS liaison or offline computing coordinator with any questions about experiment-specific scripts.