Ifdh commands » History » Version 30
Michael Kirby, 03/20/2017 10:15 AM
1 | 2 | Marc Mengel | h1. ifdh commands |
---|---|---|---|
2 | 1 | Marc Mengel | |
3 | 28 | Marc Mengel | Note that many of these commands are new in ifdh v1_3_x and/or v1_7_x. |
4 | 1 | Marc Mengel | |
5 | 28 | Marc Mengel | Many commands implicitly use a [[State Directory]]. |
6 | 28 | Marc Mengel | |
7 | 1 | Marc Mengel | h2. Environment variables |
8 | 15 | Marc Mengel | |
9 | 28 | Marc Mengel | See [[Environment Variable Reference]] |
10 | 15 | Marc Mengel | |
11 | 15 | Marc Mengel | h2. Commands |
12 | 9 | Marc Mengel | |
13 | 29 | Marc Mengel | * *ifdh* *cp* _args_ |
14 | 22 | Marc Mengel | |
15 | 22 | Marc Mengel | general file copy using cpn locks dd, gridftp, or srmcp |
16 | 22 | Marc Mengel | supports: |
17 | 22 | Marc Mengel | |
18 | 23 | Marc Mengel | * @cp src1 dest1 [';' src2 dest2 [';'...]] -- basic source/dest filenames @ |
19 | 23 | Marc Mengel | * @cp -r src1 dest1 [';' src2 dest2 [';'...]] -- recursive directory copies @ |
20 | 23 | Marc Mengel | * @cp -D src1 src2 destdir1 [';' src3 src4 destdir2 [';'...]] -- copies to dest. directory @ |
21 | 23 | Marc Mengel | * @cp -f file_with_src_space_dest_lines -- copies to a list file @ |
22 | 22 | Marc Mengel | * any of the above can take --force={cpn,gridftp,srmcp,expgridftp} |
23 | 22 | Marc Mengel | * any of the file/dest arguments can be URIs |
24 | 23 | Marc Mengel | |
25 | 1 | Marc Mengel | |
26 | 30 | Michael Kirby | Users might consider utilizing these environment variables to control the response of "ifdh cp" commands: |
27 | 30 | Michael Kirby | |
28 | 30 | Michael Kirby | * @export IFDH_GRIDFTP_EXTRA="-st N"@ which sets the stall timeout at N seconds |
29 | 30 | Michael Kirby | * @export IFDH_CP_MAXRETRIES=N@, we find 2 is usually a good number |
30 | 30 | Michael Kirby | * note that IFDH_GRIDFTP_EXTRA can be used to pass many arguments to globus-url-copy http://toolkit.globus.org/toolkit/docs/3.2/gridftp/user/globusurlcopy.html |
31 | 30 | Michael Kirby | * full environment variables here: [[Environment Variable Reference]] |
32 | 30 | Michael Kirby | |
33 | 25 | Marc Mengel | * *ifdh* *fetchInput* _src_uri_ |
34 | 22 | Marc Mengel | |
35 | 22 | Marc Mengel | get input file to local scratch, return scratch location |
36 | 22 | Marc Mengel | |
37 | 25 | Marc Mengel | * *ifdh* *localPath* _src_uri_ |
38 | 22 | Marc Mengel | |
39 | 22 | Marc Mengel | return scratch location fetchInput would give, without copying |
40 | 22 | Marc Mengel | |
41 | 24 | Marc Mengel | * *ifdh* *addOutputFile* _filename_ |
42 | 22 | Marc Mengel | |
43 | 22 | Marc Mengel | add output file to set |
44 | 22 | Marc Mengel | |
45 | 25 | Marc Mengel | * *ifdh* *copyBackOutput* _dest_dir_ |
46 | 22 | Marc Mengel | |
47 | 22 | Marc Mengel | copy output file set to destination with cpn or srmcp |
48 | 22 | Marc Mengel | |
49 | 24 | Marc Mengel | * *ifdh* *log* _args_ |
50 | 22 | Marc Mengel | |
51 | 22 | Marc Mengel | logging |
52 | 22 | Marc Mengel | |
53 | 25 | Marc Mengel | * *ifdh* *enterState* _state_ |
54 | 22 | Marc Mengel | |
55 | 22 | Marc Mengel | log entering/leaving states |
56 | 22 | Marc Mengel | |
57 | 24 | Marc Mengel | * *ifdh* *leaveState* _state_ |
58 | 22 | Marc Mengel | |
59 | 22 | Marc Mengel | |
60 | 24 | Marc Mengel | * *ifdh* *createDefinition* _name dims user group_ |
61 | 22 | Marc Mengel | |
62 | 22 | Marc Mengel | make a named dataset definition from a dimension string |
63 | 22 | Marc Mengel | |
64 | 24 | Marc Mengel | * *ifdh* *deleteDefinition* _name_ |
65 | 22 | Marc Mengel | |
66 | 22 | Marc Mengel | remove data set definition |
67 | 22 | Marc Mengel | |
68 | 24 | Marc Mengel | * *ifdh* *describeDefinition* _name_ |
69 | 22 | Marc Mengel | |
70 | 22 | Marc Mengel | describe a named dataset definition |
71 | 22 | Marc Mengel | |
72 | 24 | Marc Mengel | * *ifdh* *translateConstraints* _dims_ |
73 | 22 | Marc Mengel | |
74 | 22 | Marc Mengel | give file list for dimension string |
75 | 22 | Marc Mengel | |
76 | 24 | Marc Mengel | * *ifdh* *locateFile* _name_ |
77 | 22 | Marc Mengel | |
78 | 22 | Marc Mengel | locate a file |
79 | 22 | Marc Mengel | |
80 | 24 | Marc Mengel | * *ifdh* *getMetadata* _name_ |
81 | 22 | Marc Mengel | |
82 | 22 | Marc Mengel | get a files metadata |
83 | 22 | Marc Mengel | |
84 | 24 | Marc Mengel | * *ifdh* *dumpStation* _name what_ |
85 | 22 | Marc Mengel | |
86 | 22 | Marc Mengel | give a dump of a SAM station status |
87 | 22 | Marc Mengel | |
88 | 24 | Marc Mengel | * *ifdh* *startProject* _name station defname_or_id user group_ |
89 | 22 | Marc Mengel | |
90 | 22 | Marc Mengel | start a new file delivery project |
91 | 22 | Marc Mengel | |
92 | 24 | Marc Mengel | * *ifdh* *findProject* _name station_ |
93 | 22 | Marc Mengel | |
94 | 22 | Marc Mengel | find a started project |
95 | 22 | Marc Mengel | |
96 | 24 | Marc Mengel | * *ifdh* *establishProcess* _projecturi appname appversion location user appfamily description filelimit_ |
97 | 22 | Marc Mengel | |
98 | 22 | Marc Mengel | set yourself up as a file consumer process for a project |
99 | 22 | Marc Mengel | |
100 | 24 | Marc Mengel | * *ifdh* *getNextFile* _projecturi processid_ |
101 | 22 | Marc Mengel | |
102 | 22 | Marc Mengel | get the next file location from a project |
103 | 22 | Marc Mengel | |
104 | 24 | Marc Mengel | * *ifdh* *updateFileStatus* _projecturi processid filename status_ |
105 | 22 | Marc Mengel | |
106 | 22 | Marc Mengel | update the file status (use: transferred, skipped, or consumed) |
107 | 22 | Marc Mengel | |
108 | 24 | Marc Mengel | * *ifdh* *endProcess* _projecturi processid_ |
109 | 22 | Marc Mengel | |
110 | 22 | Marc Mengel | end the process |
111 | 22 | Marc Mengel | |
112 | 24 | Marc Mengel | * *ifdh* *dumpProject* _projecturi_ |
113 | 22 | Marc Mengel | |
114 | 22 | Marc Mengel | say what the sam station knows about your process |
115 | 22 | Marc Mengel | |
116 | 24 | Marc Mengel | * *ifdh* *setStatus* _projecturi processid status_ |
117 | 22 | Marc Mengel | |
118 | 22 | Marc Mengel | set process status |
119 | 22 | Marc Mengel | |
120 | 24 | Marc Mengel | * *ifdh* *endProject* _projecturi_ |
121 | 22 | Marc Mengel | |
122 | 22 | Marc Mengel | end the project |
123 | 22 | Marc Mengel | |
124 | 24 | Marc Mengel | * *ifdh* *cleanup* |
125 | 22 | Marc Mengel | |
126 | 22 | Marc Mengel | clean up any tmp file stuff |
127 | 22 | Marc Mengel | |
128 | 24 | Marc Mengel | * *ifdh* *renameOutput* _how_ |
129 | 22 | Marc Mengel | |
130 | 22 | Marc Mengel | give output files reported with addOutputFile a unique name |
131 | 22 | Marc Mengel | |
132 | 24 | Marc Mengel | * *ifdh* *mv* _args_ |
133 | 22 | Marc Mengel | |
134 | 22 | Marc Mengel | general file rename using mvn or srmcp |
135 | 22 | Marc Mengel | |
136 | 24 | Marc Mengel | * *ifdh* *ls* _loc recursion_depth force_ |
137 | 22 | Marc Mengel | |
138 | 22 | Marc Mengel | Get a list of directory contents, or check existence of files |
139 | 22 | Marc Mengel | |
140 | 24 | Marc Mengel | * *ifdh* *mkdir* _loc force_ |
141 | 22 | Marc Mengel | |
142 | 22 | Marc Mengel | make a directory (i.e. for file destination) |
143 | 22 | Marc Mengel | |
144 | 24 | Marc Mengel | * *ifdh* *rm* _loc force_ |
145 | 22 | Marc Mengel | |
146 | 22 | Marc Mengel | remove files |
147 | 26 | Dennis Box | |
148 | 26 | Dennis Box | * *ifdh* *rmdir* loc force |
149 | 26 | Dennis Box | |
150 | 26 | Dennis Box | remove directories |
151 | 26 | Dennis Box | |
152 | 26 | Dennis Box | * *ifdh* *more* loc |
153 | 26 | Dennis Box | |
154 | 26 | Dennis Box | view text files |
155 | 26 | Dennis Box | |
156 | 26 | Dennis Box | * *ifdh* *pin* loc seconds |
157 | 26 | Dennis Box | |
158 | 26 | Dennis Box | pin file |
159 | 26 | Dennis Box | |
160 | 26 | Dennis Box | * *ifdh* *chmod* mode loc force |
161 | 26 | Dennis Box | |
162 | 26 | Dennis Box | change file permissions |
163 | 26 | Dennis Box | |
164 | 26 | Dennis Box | * *ifdh* *rename* loc loc2 force |
165 | 26 | Dennis Box | |
166 | 26 | Dennis Box | atomic rename items in same directory/fs |
167 | 26 | Dennis Box | |
168 | 26 | Dennis Box | * *ifdh* *ll* loc recursion_depth force |
169 | 26 | Dennis Box | |
170 | 26 | Dennis Box | list files with long listing |
171 | 26 | Dennis Box | use recursion_depth== 0 to check directory without contents |
172 | 26 | Dennis Box | |
173 | 26 | Dennis Box | * *ifdh* *lss* loc recursion_depth force |
174 | 26 | Dennis Box | |
175 | 26 | Dennis Box | list files with sizes |
176 | 26 | Dennis Box | use recursion_depth== 0 to check directory without contents |
177 | 26 | Dennis Box | |
178 | 26 | Dennis Box | * *ifdh* *findMatchingFiles* path glob |
179 | 26 | Dennis Box | |
180 | 26 | Dennis Box | find filenames and sizes matching pattern |
181 | 26 | Dennis Box | |
182 | 26 | Dennis Box | * *ifdh* *fetchSharedFiles* args |
183 | 26 | Dennis Box | |
184 | 26 | Dennis Box | filenames and sizes matching pattern moved locally enough to be seen |
185 | 26 | Dennis Box | |
186 | 26 | Dennis Box | * *ifdh* *locateFiles* args |
187 | 26 | Dennis Box | |
188 | 26 | Dennis Box | locate multiple files |
189 | 26 | Dennis Box | |
190 | 26 | Dennis Box | * *ifdh* *checksum* loc |
191 | 26 | Dennis Box | |
192 | 26 | Dennis Box | cheksum file |