Project

General

Profile

SupportedProtocols

Each of the protocols listed below with their URL prefix is supported in idfh for ifdh cp, ls, etc. You can give full URLs as source and/or destination, or you can use --force=protocol (or set IFDH_FORCE=protocol in the environment) to require that specific protocol be used to access various endpoints. This is recommended only when you (really really) need to work around some problem or permissions issue; generally we recommend letting ifdh pick what it thinks is the best protocol for a given endpoint path, or spelling out the full URL for a given endpoint using that protocol, as this leaves your scripts open to being given other endpoints later that you may have not planned for initially.

expgridftp

This is a --force=option that isn't a protocol, rather it specified both the gsiftp: protocol and using special per-experiment GridFTP servers for access to BlueArc to get different file permissions mapping. Now that we are running improved GUMS and have appropriate role/experimenter based mappings, this is no longer needed, and is basically an alias for gridftp/gsiftp.

gsiftp: GridFTP

This is the workhorse protocol preferred for most tasks in the current systems. It is an X509/Grid Proxy authenticated FTP service, which ifdh currently supports using globus-url-copy for file transfers, and uberftp for other operations (directory listings, chmod, mkdir, etc.) from the osg-client bundle.

srm:

This is the storage management protocol used by many Grid storage elements; it is X509 or Grid Proxy authenticated. Current implementations chat with you and then assign you a gridftp url to use, so it's nearly always slower than direct gsiftp:, but for a random grid storage element it may provide load balancing, etc. that a direct gsiftp: url might not. ifdh currently supports this using lcg-cp for file transfers, and the srmls, srmmkdir, etc. tools for other operations; or in more recent versions "gfal" if it's available (osg-client 3.2 or later).

In DCache, the srm: interface will give you directory listings of other experiments' areas if they are world readable, where the gsiftp: interface will not.

http:, https: Hypertext Transfer Protocol

This is the protocol you're familiar with from your favorite web browser. The https: form is also X509 or Grid Proxy authenticated. You can use it to talk to DCache (i.e. https://fndca4a.fnal.gov:2880/pnfs/fnal.gov/usr/$EXPERIMENT/...) , or to transfer data from other web services. For posting data, it uses the WebDAV extensions to http. Currently ifdh uses curl for file transfers, and complains of not supporting mkdir, chmod, etc.

ucondb:

This is a shorthand for a longer http: URL used for our new web-based conditions database here at Fermi.

s3:

This is the protocol for dealing with Amazon's s3 cloud storage; it's supported using the "aws" command line script bundle from Amazon. It assumes you have gotten your Amazon environment and authentication setup independently. (see getawscreds/useawscreds in the fife_utils package for how to do this in non Amazon grid jobs.).

root:, xroot: XRootd

This is the protocol for streaming root files, which can also do binary transfers of non-root files.
In older ifdh versions, it's only supported by fetchInput calls, which pass the full url along to
the application, assuming it knows how to use them.

ifdh in v1_8_11 and later supports this using the "xrdcp" command for transfers, and "xrdfs" for others.

i: IRODS

This is the IRODS location independent file transfer layer; ifdh supports this using the irods utilities (icp ils, ichmod, etc.). It assumes you have gotten your irods environment and authentication setup independently. (See http://irods.org/wp-content/uploads/2015/06/GettingStartedwiRODS4.1.pdf)