Project

General

Profile

File Transfer Service (FTS) for Offline

EXPERT USE ONLY!

For the offline production we have an instance of the File Transfer Service (FTS) which was developed for the DAQ systems to move and catalog data files.

The way the FTS works, is that it looks in a pre-defined "dropbox" area for new files that match different types of files that are registered with the service (i.e. data files, monte carlo files, log files etc...). When it finds a new file in this area it attempts to catalog the file by extracting key metadata from the file and then declaring the file to the SAM data handling system.

Once the system is declared to SAM it is moved to an appropriate set of locations (i.e. well defined areas on disk or tape) where it can subsequently be located and retrieved by the SAM system when running jobs.

Checking FTS Status

The status of the FTS system can be checked by looking at a status display provided by the FTS. There are multiple different FTS instances. All of our files are load balanced across all of the instances.

For production processing, there are three FTS instances, each running on separate . Each of them scans a subset of the top level hex directories

Hex directories Hostname Monitoring URL Running Directory
0-3 novasamgpvm01.fnal.gov http://novasamgpvm01.fnal.gov:8888/fts/status /nova/app/home/novapro/NovaOfflineFTS/fts_rundir-novasamgpvm01.fnal.gov
4-7 novasamgpvm02.fnal.gov http://novasamgpvm02.fnal.gov:8888/fts/status /nova/app/home/novapro/NovaOfflineFTS/fts_rundir-novasamgpvm02.fnal.gov
8-b novasamgpvm03.fnal.gov http://novasamgpvm03.fnal.gov:8888/fts/status /nova/app/home/novapro/NovaOfflineFTS/fts_rundir-novasamgpvm03.fnal.gov
c-f novasamgpvm04.fnal.gov http://novasamgpvm04.fnal.gov:8888/fts/status /nova/app/home/novapro/NovaOfflineFTS/fts_rundir-novasamgpvm04.fnal.gov

If the status page has problems loading this probably indicates that the FTS is hung up.

If the status page loads then you can see what is happening with files and even retry failed files etc...

Off-site users must find a way to pretend they are on-site in order to see these pages. An easy way to do this is using Fermilab's VPN services. Another solution involves an SSH tunnel and a SOCKS proxy, which serves as an alternative for when the VPN option is prohibitive.

Restarting FTS

The FTS runs on novasamgpvm01(2).fnal.gov under the "novapro" account. You should be able to login there if you are in the k5login file for the novapro account.

To restart the FTS you need to login to the respective SAM VM, Where XX is the number of the novasamgpvm machine (currently 01, 02, 03).

ssh novapro@novasamgpvmXX.fnal.gov
source ~/NovaOfflineFTS/setup.sh 
setup FileTransferService

Then you will have access to the start/stop fts scripts.

You can stop/start the daemon with:

stop_fts ~/NovaOfflineFTS/fts_rundir-novasamgpvmXX.fnal.gov/

or
start_fts ~/NovaOfflineFTS/fts_rundir-novasamgpvmXX.fnal.gov ~/NovaOfflineFTS/fts_rundir-novasamgpvmXX.fnal.gov/fts_config.ini

The config directory is fts_rundir and the standard config file is fts_config_vmXX.ini. These config files are being version controlled in svn. The package is NovaFTS.

Alternatively it is now possible to update the config files, and restart the fts for all of the instances of the offline in one step from any machine.
From the NovaFTS package in a development test release follow these steps.

1) Update the config file in the balanced subdirectory
2) Commit your changes to the config file.
3) Run the script restart_all.sh (./restart_all.sh)

restart_all.sh calls update_fts_config.sh and restart_fts.sh directly on the machines that run the fts. These scripts update a development test release owned by novapro. Then starts and stops the fts for each instance. You must be in the novapro k5login file, to do this other wise you wont be able to ssh into the novapro account.

File Path Construction

The destination paths in the FTS support expansion of the different metadata variables which are extracted from the files. To test how these expansions will get evaluated you can use a combination of samweb and the check_path_template to expand and replace a path based on a given file.

To do this:

samweb -e nova get-metadata --json fardet_r00014072_s53_t02_S14-03-20_v1_data.artdaq.root | check_path_template 'enstore:/pnfs/nova/production/raw2root/${DAQ2RawDigit.base_release}/${Online.Detector}/${run_number/100[6]}/${run_number}/${Online.Stream[2]}'

You will get the output:

enstore:/pnfs/nova/production/raw2root/S14-03-20/fardet/000140/14072/02

Alternatively you can store the json formated metadata in a file and then check the template that way:

samweb -e nova get-metadata --json fardet_r00014072_s53_t02_S14-03-20_v1_data.artdaq.root > metadata_file.json
check_path_template 'enstore:/pnfs/nova/production/raw2root/${DAQ2RawDigit.base_release}/${Online.Detector}/${run_number/100[6]}/${run_number}/${Online.Stream[2]}' metadata_file.json

Adding A Metadata Parameter

Sometimes you will need to add a new parameter to the SAM database in order to upload your files. To do this:

samweb add-parameter <parameter name> <parameter type>

Example:
For a parameter called "simulated.singlepused" which has a type of "string"
samweb add-parameter simulated.singlepused string

To list current parameters:

samweb list-parameters