Project

General

Profile

Creating a New Data Tier

This page explains the steps to create a new data tier, including registering it with SAM and updating the FTS and tape-backed storage infrastructure to support it. This is required whenever a new data tier is introduced into a production workflow. Note that these actions generally require you to be logged in as uboonepro.

Register the Data Tier with SAM

Data tiers are required to be registered with SAM. You can register your new data tier using the following line. Note that there is a 25-character limit on the length of the data tier name.

samweb add-value data_tiers [new data tier name] [new data tier description]

Create New Sub-Directories

Each data tier requires its own directories in the FTS dropbox areas and in the tape-backed storage area. You need to create a new sub-directory with the name of the data tier in each of the following locations. Choose data or mc as appropriate.

  • /pnfs/uboone/persistent/uboonepro/dropbox/{data, mc}/uboone/
  • /pnfs/uboone/scratch/uboonepro/dropbox/{data, mc}/uboone/
  • /pnfs/uboone/{data, mc}/uboone/

Modify the File Family Tag for the Tape-Backed Directory

Tape-backed directories require certain information, including a tag describing the file family stored in that directory. Go into the directory you just created within /pnfs/uboone/{data, mc}/uboone/. Change the file family tag using the following line.

echo "[new data tier file family]" > ".(tag)(file_family)" 

You can check that this worked as expected (and/or check the file_family tag for similar data tiers to help you decide what to use for yours) by executing the following line in the directory of interest.

grep "" $(cat ".(tags)()")

If you are also creating a new file family, you will need to open a ticket in Service Now to update the SFA configuration.

Additional documentation is available on the dCache wiki page.

Modify the FTS Script

The FTS script needs to be modified such that it will scan the newly-created dropbox directories. These changes need to be made on the FTS server, uboonesamgpvm01. You can log in to that machine using the following line executed from a uboone gpvm.

ssh -l uboonepro uboonesamgpvm01

The script you need to edit is FTS/uboonesamgpvm01/uboonesamgpvm01_fts_config.ini. Open this with a text editor and add the dropbox directories you created to the appropriate scan-dirs block.

After updating the script, you will need to restart the FTS for your changes to take effect. You can do that using the following lines executed from the ~/FTS/uboonesamgpvm01 directory on uboonesamgpvm01.

source setup_fts.sh
stop_fts .
./fts_atboot.sh

Additional documentation is available at the FTS wiki page.

Check if data tier is registered correctly

You could monitor in real time the progress of FTS here at

/uboone/app/home/uboonepro/FTS/uboonesamgpvm01/fts_logs/fts_ub.2018-XX-XX.log

Make sure you see the newly added directories are being scanned by FTS after you restart the FTS.

Update the dCache Wiki Page

Update the table on the dCache wiki page to include your new data tier. (You will need to be logged in to edit the page, which you can do by clicking on "Sign in" at the upper right and entering your Fermilab Services username and password.)

P.S. While you're at it, update this wiki page too!