Project

General

Profile

Ifdh commands » History » Version 21

Marc Mengel, 05/05/2014 02:26 PM

1 2 Marc Mengel
h1. ifdh commands
2 1 Marc Mengel
3 15 Marc Mengel
Note that many of these commands are new in ifdh v1_3_x.
4 15 Marc Mengel
5 15 Marc Mengel
h2. Environment variables
6 15 Marc Mengel
7 15 Marc Mengel
* IFDH_DEBUG=1 for voluminous debug output
8 15 Marc Mengel
* EXPERIMENT set to an experiment name ("minerva", "nova", etc.)
9 15 Marc Mengel
  if it can't be guessed from your group,etc.
10 15 Marc Mengel
* IFDH_BASE_URI to pick a particular samweb server.
11 15 Marc Mengel
* IFDH_GRIDFTP_EXTRA, IFDH_SRM_EXTRA, IFDH_DD_EXTRA flags to set options
12 15 Marc Mengel
  flor globus-url-copy, lcg_cp, dd, etc.
13 15 Marc Mengel
* IFDH_WEB_TIMEOUT for samweb timeouts.
14 15 Marc Mengel
15 15 Marc Mengel
h2. Commands
16 9 Marc Mengel
17 7 Marc Mengel
* *ifdh* *cp* _src  dest_
18 2 Marc Mengel
** general file copy using cpn or srmcp
19 10 Marc Mengel
@ifdh cp [--force=srm|gridftp|cpn|expftp] [-r]  src1 dest1 [ ";" src2 ...  dest2 [ ";" ... ]  ]@
20 10 Marc Mengel
@ifdh cp [--force=srm|gridftp|cpn|expftp] -f listfile@
21 10 Marc Mengel
@ifdh cp [--force=srm|gridftp|cpn|expftp] -D [-r]  src1 src2 destdir1 [ ";" src3 src4 ...  destdir2 [ ";" ... ]  ]@
22 10 Marc Mengel
One can also set IFDH_FORCE= in the environment rather than using --force=
23 21 Marc Mengel
Note that -r means recursive copy of directories, and -D means the destination is a directory.
24 21 Marc Mengel
25 12 Marc Mengel
* *ifdh* *ls* _dest_ [ _depth_ [--force=srm|gridftp|cpn]]
26 11 Marc Mengel
** list file or directory 
27 13 Marc Mengel
** also confirms existence
28 11 Marc Mengel
* *ifdh* *mkdir* _dest_ [--force=srm|gridftp|cpn]
29 11 Marc Mengel
** make directories
30 11 Marc Mengel
* *ifdh* *rm* _dest_ [--force=srm|gridftp|cpn]
31 11 Marc Mengel
** remove files
32 11 Marc Mengel
* *ifdh* *rmdir* _dest_ [--force=srm|gridftp|cpn]
33 11 Marc Mengel
** remove directories
34 7 Marc Mengel
* *ifdh* *fetchInput* _src_uri_
35 2 Marc Mengel
** get input file to local scratch, return scratch location
36 7 Marc Mengel
* *ifdh* *addOutputFile* _filename_
37 2 Marc Mengel
** add output file to set
38 16 Marc Mengel
* *ifdh* *renameOutput* _how_
39 16 Marc Mengel
** rename output files, _how_ is one of
40 16 Marc Mengel
*** unique -- add time,host,pid text to filename to make it unique
41 16 Marc Mengel
*** s/this/that/ -- convert nearest input filename to output filename with substitution, now takes full perl substitution/regexp
42 20 Marc Mengel
*** exec:/some/script -- use a [[RenameScript]] to rename the files
43 7 Marc Mengel
* *ifdh* *copyBackOutput* _dest_dir_
44 2 Marc Mengel
** copy output file set to destination with cpn or srmcp
45 7 Marc Mengel
* *ifdh* *log* _message_
46 2 Marc Mengel
** logging 
47 7 Marc Mengel
* *ifdh* *enterState* _state_
48 2 Marc Mengel
** log entering/leaving states
49 7 Marc Mengel
* *ifdh* *leaveState* _state_
50 2 Marc Mengel
** log entering/leaving states
51 7 Marc Mengel
* *ifdh* *createDefinition* _name  dims  user  group_
52 2 Marc Mengel
** make a named dataset definition from a dimension string
53 7 Marc Mengel
* *ifdh* *deleteDefinition* _name_
54 2 Marc Mengel
** remove data set definition
55 7 Marc Mengel
* *ifdh* *describeDefinition* _name_
56 2 Marc Mengel
** describe a named dataset definition
57 7 Marc Mengel
* *ifdh* *translateConstraints* _dims_
58 2 Marc Mengel
** give file list for dimension string
59 7 Marc Mengel
* *ifdh* *locateFile* _name_
60 2 Marc Mengel
** locate a file
61 7 Marc Mengel
* *ifdh* *getMetadata* _name_
62 2 Marc Mengel
** get a files metadata
63 7 Marc Mengel
* *ifdh* *dumpStation* _name  what_
64 2 Marc Mengel
** give a dump of a SAM station status
65 7 Marc Mengel
* *ifdh* *startProject* _name  station  defname_or_id  user  group_
66 2 Marc Mengel
** start a new file delivery project
67 7 Marc Mengel
* *ifdh* *findProject* _name  station_
68 2 Marc Mengel
** find a started project
69 7 Marc Mengel
* *ifdh* *establishProcess* _projecturi  appname  appversion  location  user  appfamily   description   filelimit_
70 2 Marc Mengel
** set yourself up as a file consumer process for a project
71 7 Marc Mengel
* *ifdh* *getNextFile* _projecturi  processid_
72 2 Marc Mengel
** get the next file location from a project
73 7 Marc Mengel
* *ifdh* *updateFileStatus* _projecturi  processid  filename  status_
74 2 Marc Mengel
** update the file status (use: transferred, skipped, or consumed)
75 7 Marc Mengel
* *ifdh* *endProcess* _projecturi  processid_
76 2 Marc Mengel
** end the process
77 14 Marc Mengel
* *ifdh* *dumpProject* _projecturi_
78 2 Marc Mengel
** say what the sam station knows about your process
79 7 Marc Mengel
* *ifdh* *setStatus* _projecturi  processid  status_
80 2 Marc Mengel
** set process status
81 7 Marc Mengel
* *ifdh* *endProject* _projecturi_
82 2 Marc Mengel
** end the project
83 8 Marc Mengel
* *ifdh* *cleanup*
84 2 Marc Mengel
** clean up any tmp file stuff