Scientific Data Management » REX » IF Data Handling Client Tools (ifdhc)

h1. ifdh commands

Note that many of these commands are new in ifdh v1_3_x and/or v1_7_x.

Many commands implicitly use a [[State Directory]].

h2. Environment variables

See [[Environment Variable Reference]]

h2. Commands

* *ifdh* *cp* _args_

	 general file copy using cpn locks dd, gridftp, or srmcp 

	 supports:  

	 * @cp src1 dest1 [';' src2 dest2 [';'...]]                     -- basic source/dest filenames @

	 * @cp -r src1 dest1 [';' src2 dest2 [';'...]]                  -- recursive directory copies @

	 * @cp -D src1 src2 destdir1 [';' src3 src4 destdir2 [';'...]]  -- copies to dest. directory @

	 * @cp -f file_with_src_space_dest_lines                        -- copies to a list file @

	 * any of the above can take --force={cpn,gridftp,srmcp,expgridftp} 

	 * any of the file/dest arguments can be URIs 

	 Users might consider utilizing these environment variables to control the response of "ifdh cp" commands:

	 * @export IFDH_GRIDFTP_EXTRA="-st N"@ which sets the stall timeout at N seconds IFDH_XROOTD_EXTRA

	 * @export IFDH_CP_MAXRETRIES=N@, we find 2 is usually a good number

	 * note that IFDH_GRIDFTP_EXTRA can be used to pass many arguments to globus-url-copy http://toolkit.globus.org/toolkit/docs/3.2/gridftp/user/globusurlcopy.html

	 * note that IFDH_XROOTD_EXTRA can be used to pass many arguments to xrootd/xrdcp actions https://linux.die.net/man/1/xrdcp 

	 * full environment variables here: [[Environment Variable Reference]]

* *ifdh* *fetchInput* _src_uri_

	 get input file to local scratch, return scratch location

* *ifdh* *localPath* _src_uri_

	 return scratch location fetchInput would give, without copying

* *ifdh* *addOutputFile* _filename_

	 add output file to set

* *ifdh* *copyBackOutput* _dest_dir_

	 copy output file set to destination with cpn or srmcp

* *ifdh* *log* _args_

	 logging 

* *ifdh* *enterState* _state_

	 log entering/leaving states

* *ifdh* *leaveState* _state_

* *ifdh* *createDefinition* _name  dims  user  group_

	 make a named dataset definition from a dimension string

* *ifdh* *deleteDefinition* _name_

	 remove data set definition

* *ifdh* *describeDefinition* _name_

	 describe a named dataset definition

* *ifdh* *translateConstraints* _dims_

	 give file list for dimension string

* *ifdh* *locateFile* _name_

	 locate a file

* *ifdh* *getMetadata* _name_

	 get a files metadata

* *ifdh* *dumpStation* _name  what_

	 give a dump of a SAM station status

* *ifdh* *startProject* _name  station  defname_or_id  user  group_

	 start a new file delivery project

* *ifdh* *findProject* _name  station_

	 find a started project

* *ifdh* *establishProcess* _projecturi  appname  appversion  location  user  appfamily   description   filelimit_

	 set yourself up as a file consumer process for a project

* *ifdh* *getNextFile* _projecturi  processid_

	 get the next file location from a project

* *ifdh* *updateFileStatus* _projecturi  processid  filename  status_

	 update the file status (use: transferred, skipped, or consumed)

* *ifdh* *endProcess* _projecturi  processid_

	 end the process

* *ifdh* *dumpProject* _projecturi_

	 say what the sam station knows about your process

* *ifdh* *setStatus* _projecturi  processid  status_

	 set process status

* *ifdh* *endProject* _projecturi_

	 end the project

* *ifdh* *cleanup* 

	 clean up any tmp file stuff

* *ifdh* *renameOutput* _how_

	 give output files reported with addOutputFile a unique name

* *ifdh* *mv* _args_

	 general file rename using mvn or srmcp

* *ifdh* *ls* _loc  recursion_depth  force_

	 Get a list of directory contents, or check existence of files

* *ifdh* *mkdir* _loc  force_

	 make a directory (i.e. for file destination)

* *ifdh* *rm* _loc  force_

	 remove files

* *ifdh* *rmdir* loc force

         remove directories

*  *ifdh* *more*  loc 

	 view text files

* *ifdh* *pin*  loc  seconds 

	 pin file

*  *ifdh* *chmod*  mode  loc  force  

	 change file permissions

* *ifdh* *rename*  loc  loc2  force  

	 atomic rename items in same directory/fs

*  *ifdh* *ll*  loc  recursion_depth  force  

	 list files with long listing 

	 use recursion_depth== 0 to check directory without contents

*  *ifdh* *lss*  loc  recursion_depth  force  

	 list files with sizes 

	 use recursion_depth== 0 to check directory without contents

*	*ifdh* *findMatchingFiles*  path  glob 

	 find filenames and sizes matching pattern

*	*ifdh* *fetchSharedFiles* args

	 filenames and sizes matching pattern moved locally enough to be seen

*	*ifdh* *locateFiles* args

	 locate multiple files

*	*ifdh* *checksum*  loc 

	 cheksum file