Ifdh commands » History » Version 16

Version 15 (Marc Mengel, 01/10/2014 03:18 PM) → Version 16/33 (Marc Mengel, 03/06/2014 02:54 PM)

h1. ifdh commands

Note that many of these commands are new in ifdh v1_3_x.

h2. Environment variables

* IFDH_DEBUG=1 for voluminous debug output
* EXPERIMENT set to an experiment name ("minerva", "nova", etc.)
if it can't be guessed from your group,etc.
* IFDH_BASE_URI to pick a particular samweb server.
flor globus-url-copy, lcg_cp, dd, etc.
* IFDH_WEB_TIMEOUT for samweb timeouts.

h2. Commands

* *ifdh* *cp* _src dest_
** general file copy using cpn or srmcp
@ifdh cp [--force=srm|gridftp|cpn|expftp] [-r] src1 dest1 [ ";" src2 ... dest2 [ ";" ... ] ]@
@ifdh cp [--force=srm|gridftp|cpn|expftp] -f listfile@
@ifdh cp [--force=srm|gridftp|cpn|expftp] -D [-r] src1 src2 destdir1 [ ";" src3 src4 ... destdir2 [ ";" ... ] ]@
One can also set IFDH_FORCE= in the environment rather than using --force=
* *ifdh* *ls* _dest_ [ _depth_ [--force=srm|gridftp|cpn]]
** list file or directory
** also confirms existence
* *ifdh* *mkdir* _dest_ [--force=srm|gridftp|cpn]
** make directories
* *ifdh* *rm* _dest_ [--force=srm|gridftp|cpn]
** remove files
* *ifdh* *rmdir* _dest_ [--force=srm|gridftp|cpn]
** remove directories
* *ifdh* *fetchInput* _src_uri_
** get input file to local scratch, return scratch location
* *ifdh* *addOutputFile* _filename_
** add output file to set
* *ifdh* *renameOutput* _how_
** rename output files, _how_ is one of
*** unique -- add time,host,pid text to filename to make it unique
*** s/this/that/ -- convert nearest input filename to output filename with substitution, now takes full perl substitution/regexp
*** exec:/some/script -- run /some/script filename list file as its first argument, file has lines of the form: @currentname inputfilename@, script should update the file to maintani currentname so copyBackOuput will work.
* *ifdh*
*copyBackOutput* _dest_dir_
** copy output file set to destination with cpn or srmcp
* *ifdh* *log* _message_
** logging
* *ifdh* *enterState* _state_
** log entering/leaving states
* *ifdh* *leaveState* _state_
** log entering/leaving states
* *ifdh* *createDefinition* _name dims user group_
** make a named dataset definition from a dimension string
* *ifdh* *deleteDefinition* _name_
** remove data set definition
* *ifdh* *describeDefinition* _name_
** describe a named dataset definition
* *ifdh* *translateConstraints* _dims_
** give file list for dimension string
* *ifdh* *locateFile* _name_
** locate a file
* *ifdh* *getMetadata* _name_
** get a files metadata
* *ifdh* *dumpStation* _name what_
** give a dump of a SAM station status
* *ifdh* *startProject* _name station defname_or_id user group_
** start a new file delivery project
* *ifdh* *findProject* _name station_
** find a started project
* *ifdh* *establishProcess* _projecturi appname appversion location user appfamily description filelimit_
** set yourself up as a file consumer process for a project
* *ifdh* *getNextFile* _projecturi processid_
** get the next file location from a project
* *ifdh* *updateFileStatus* _projecturi processid filename status_
** update the file status (use: transferred, skipped, or consumed)
* *ifdh* *endProcess* _projecturi processid_
** end the process
* *ifdh* *dumpProject* _projecturi_
** say what the sam station knows about your process
* *ifdh* *setStatus* _projecturi processid status_
** set process status
* *ifdh* *endProject* _projecturi_
** end the project
* *ifdh* *cleanup*
** clean up any tmp file stuff