Project

General

Profile

File Transfer Service Information

Overview

The File Transfer Service (FTS) is a daemon process that is designed to automate the transfer of files from one storage system to another. It can handle local data transfers, transfers to remote nodes, and third party transfers between remote nodes. FTS works in conjunction with the SAM data cataloguing system to track the location of transferred files. FTS can also be configured to manage the disposition of input files after the transfers have been successfully completed or after the transfer attempts have failed a specified number of retries. Normal user interaction with FTS consists of simply placing the files to be transferred into a designated dropbox area.

Operation of FTS is controlled by a configuration file which allows designation of dropbox areas, dropbox scanning intervals, stored file disposition, transfer destinations, and file types to be considered for transfer.

User monitoring of FTS is available via a web page which tracks current and past transfers, transfer rates, and the operational status of critical underpinning components.

FTS Details

Installation

The FTS is distributed as a ups product. The latest version of FTS should be installed unless it is known that an older version is needed for compatibility reasons. The versions of FTS available can be seen with upd:

$ upd list -aK+ FileTransferService

FTS depends on having a properly configured sam client product on the local machine. 'setup sam; sam ping dbserver' should return the name of the appropriate SAM DB server for the relevant experiment. If multiple sam configurations exist on the local machine then the appropriate qualifier should be used with the setup commands. This can be done either directly on the 'setup sam' command or indirectly when the FTS product itself is setup:

$ setup sam -q fts-sam-config     (where fts-sam-config is the name of the appropriate sam configuration)

or

$ setup FileTransferService v3_11_1 -q fts-sam-config

If data files are required to be copied anywhere then FTS requires a SAM FSS server (this is not required to be running on the local machine), a SAM stager running locally, and the sam_cp product installed and configured for the needed transfer methods. If files are going to be archived to tape then the pnfs file system must be installed on the machine, and the encp product installed. If the small file aggregation (SFA) service is being used in the archive then the encp version must be v3_11 or later.

External dependencies

SLF6 systems require the openssl098e compatibility RPM be installed.

Recommended Directory Structure

FTS requires access to a configuration file, certificates for authentication, log files, and various optional plugin files. A typical installation might look like:

FTS_rundir           This directory should contain the FTS configuration file.   When FTS starts it will place
                     in this directory a file named 'fts.pid' which will contain the process ID of the running
                     FTS process.
FTS_rundir/cert      This directory should contain the certificates and keys that FTS will use to authenticate
                     itself. Certificate files should have permissions set to 644.  Key files must have 
                     permissions set to 600.
FTS_rundir/fts_logs  This directory will contain the log files that the FTS process generates.  This location
                     must be specified in the FTS configuration file.
FTS_rundir/plugins   This directory will contain any plugins used by FTS.  This location must be specified in
                     the configuration file if optional plugins are used by FTS.

The UID under which the FTS process runs should have ownership of the above directory structure. It must have write access to both FTS_rundir and FTS_rundir/fts_logs.

Starting and Stopping FTS

Two scripts are provided with the FTS product to facilitate starting and stopping the service. These scripts, start_fts and stop_fts, reside in $FILETRANFERSERVICE/bin. This directory will be in your path after setting up the FileTransferService product.

In addition to the scripts provided by FTS, it is recommended that all installation have a local setup script named "setup_fts.sh" in the FTS_rundir area. This script should establish the location of the UPS database to be used and also set up any products necessary for FTS operation. A typical setup_fts.sh script might contain the following:

#! /bin/bash

source /grid/fermiapp/products/FTS/etc/setups.sh
setup sam -q my_config
setup FileTransferService v3_11_1

In the above "my_config" is the name of the appropriate SAM configuration for your experiment. It is best to set up an explicit version of FTS rather than to rely on the UPS declaration of which version is current.

To start FTS first make sure you are logged in under an appropriate account. As noted above, this account must have write access to the FTS_rundir
and fts_logs directories. It must also have read access to any files which it is expected to transfer. The account must also have write access to any scanned directories which contain files for which post-transfer processing is specified in the FTS configuration file.

Start FTS by sourcing the user supplied local setup script and then executing the start_fts script supplied by the FTS product:

$ cd /path_to/FTS_rundir
$ source setup_fts.sh
$ start_fts /path_to/FTS_rundir fts_config.ini

where fts_config.ini is the FTS configuration file. This configuration file is assumed to be in the FTS_rundir area. If the configuration file is in a different directory then the full absolute path to the file should be given.

To stop FTS do the following:

$ cd /path_to/FTS_rundir
$ source setup_fts.sh
$ stop_fts /path_to/FTS_rundir

Configuration

The operation of FTS is controlled by parameters that are set in a configuration file. The configuration file is read when the FTS daemon is started. The FTS daemon must be restarted for any changes in the configuration file to take effect. A brief description of the available parameters can be found here.

The configuration file follows the standard INI file syntax as described here. Explicit details of how the configuration file is parsed can be found in the python documentation for SafeConfigParser.

There are three section types allowed in the configuration file:
  1. [main] This section contains parameters that effect global operation of FTS. There is usually only one [main] section in the configuration file.
  2. [filetype <type>] The filetype sections contain configuration parameters that only effect action of FTS on the specified filetype. There may be any number of [filetype] sections in the configuration file.
  3. [plugin <plugin name>] The plugin sections contain configuration parameters that only effect the operation of the specified plugin. There may be any number of plugin sections in the configuration file.

If multiple instances of any identical section types exist in the configuration file, then the values in the last instance in the file take precedence for any duplicated parameters.

Configuration Details
[main]

Parameter Description Default
log-file This is the path and prefix of the log file. The prefix will have the date and .log appended in the format .YYYY-dd-mm.log. The hostname may be included in the log file name with the pattern ${hostname}. For example, the setting 'log-file = /path_to/FTS_rundir/fts_logs/fts_${hostname}' will produce a log file in /path_to/FTS_rundir/fts_logs with a name like fts_samgpvm01.2014-06-12.log if the FTS daemon is running on node samgpvm01.fnal.gov. If the log-file parameter is not specified then a log named 'twistd.log' is produced in the FTS run directory.
filetypes This is a blank separated list of the filetype sections in the configuration file to activate. No default. If no filetype sections are specified then no files will be transferred.
samweb-url The URL of the samweb server to be used for file metadata information. A list of available servers can be found here. The authenticated server instances should be used. No default. A samweb server instance must be specified.
x509-client-certificate This is the path to an x509 certificate to be used to authenticate to the samweb server. The user name under which the FTS daemon runs must be a valid user in the samweb database and the subject of this certificate must be attached to that user name in the data base. The subject can be added with the 'samweb modify-user --addgridsubject' command. If this parameter is not specified FTS will use the $X509_USER_CERT environment variable for the certificate location. If $X509_USER_CERT is not defined then FTS will check $X509_USER_PROXY. If neither $X509_USER_CERT nor $X509_USER_PROXY is defined then FTS will look for a certificate in /tmp/x509up_u<user id>.
x509-client-key This is the path to an x509 key file associated with the x509-client-certificate above. Since FTS operates as a daemon process, this key must not be password protected. If this parameter is not specified FTS will use the $X509_USER_KEY environment variable for the key location. If neither the parameter is set nor the environment variable defined, then FTS will use the x509-client-certificate file.
sam-fss-name Name of the SAM station associated with the fss instance used for storing files. The name 'fake-fss' may be used to do rudimentary testing of the FTS installation if a SAM station/fss combination is not yet running. No default. This must be specified.
transfer-retries This is the number of times FTS will retry a transfer before declaring it a failure. Default is infinite.

Starting and stopping it