Ifdh commands » History » Version 31
Michael Kirby, 03/20/2017 10:35 AM
1 | 2 | Marc Mengel | h1. ifdh commands |
---|---|---|---|
2 | 1 | Marc Mengel | |
3 | 28 | Marc Mengel | Note that many of these commands are new in ifdh v1_3_x and/or v1_7_x. |
4 | 1 | Marc Mengel | |
5 | 28 | Marc Mengel | Many commands implicitly use a [[State Directory]]. |
6 | 28 | Marc Mengel | |
7 | 1 | Marc Mengel | h2. Environment variables |
8 | 15 | Marc Mengel | |
9 | 28 | Marc Mengel | See [[Environment Variable Reference]] |
10 | 15 | Marc Mengel | |
11 | 15 | Marc Mengel | h2. Commands |
12 | 9 | Marc Mengel | |
13 | 29 | Marc Mengel | * *ifdh* *cp* _args_ |
14 | 22 | Marc Mengel | |
15 | 22 | Marc Mengel | general file copy using cpn locks dd, gridftp, or srmcp |
16 | 22 | Marc Mengel | supports: |
17 | 22 | Marc Mengel | |
18 | 23 | Marc Mengel | * @cp src1 dest1 [';' src2 dest2 [';'...]] -- basic source/dest filenames @ |
19 | 23 | Marc Mengel | * @cp -r src1 dest1 [';' src2 dest2 [';'...]] -- recursive directory copies @ |
20 | 23 | Marc Mengel | * @cp -D src1 src2 destdir1 [';' src3 src4 destdir2 [';'...]] -- copies to dest. directory @ |
21 | 23 | Marc Mengel | * @cp -f file_with_src_space_dest_lines -- copies to a list file @ |
22 | 22 | Marc Mengel | * any of the above can take --force={cpn,gridftp,srmcp,expgridftp} |
23 | 22 | Marc Mengel | * any of the file/dest arguments can be URIs |
24 | 23 | Marc Mengel | |
25 | 1 | Marc Mengel | |
26 | 30 | Michael Kirby | Users might consider utilizing these environment variables to control the response of "ifdh cp" commands: |
27 | 30 | Michael Kirby | |
28 | 31 | Michael Kirby | * @export IFDH_GRIDFTP_EXTRA="-st N"@ which sets the stall timeout at N seconds IFDH_XROOTD_EXTRA |
29 | 30 | Michael Kirby | * @export IFDH_CP_MAXRETRIES=N@, we find 2 is usually a good number |
30 | 1 | Marc Mengel | * note that IFDH_GRIDFTP_EXTRA can be used to pass many arguments to globus-url-copy http://toolkit.globus.org/toolkit/docs/3.2/gridftp/user/globusurlcopy.html |
31 | 31 | Michael Kirby | * note that IFDH_XROOTD_EXTRA can be used to pass many arguments to xrootd/xrdcp actions https://linux.die.net/man/1/xrdcp |
32 | 30 | Michael Kirby | * full environment variables here: [[Environment Variable Reference]] |
33 | 30 | Michael Kirby | |
34 | 25 | Marc Mengel | * *ifdh* *fetchInput* _src_uri_ |
35 | 22 | Marc Mengel | |
36 | 22 | Marc Mengel | get input file to local scratch, return scratch location |
37 | 22 | Marc Mengel | |
38 | 25 | Marc Mengel | * *ifdh* *localPath* _src_uri_ |
39 | 22 | Marc Mengel | |
40 | 22 | Marc Mengel | return scratch location fetchInput would give, without copying |
41 | 22 | Marc Mengel | |
42 | 24 | Marc Mengel | * *ifdh* *addOutputFile* _filename_ |
43 | 22 | Marc Mengel | |
44 | 22 | Marc Mengel | add output file to set |
45 | 22 | Marc Mengel | |
46 | 25 | Marc Mengel | * *ifdh* *copyBackOutput* _dest_dir_ |
47 | 22 | Marc Mengel | |
48 | 22 | Marc Mengel | copy output file set to destination with cpn or srmcp |
49 | 22 | Marc Mengel | |
50 | 24 | Marc Mengel | * *ifdh* *log* _args_ |
51 | 22 | Marc Mengel | |
52 | 22 | Marc Mengel | logging |
53 | 22 | Marc Mengel | |
54 | 25 | Marc Mengel | * *ifdh* *enterState* _state_ |
55 | 22 | Marc Mengel | |
56 | 22 | Marc Mengel | log entering/leaving states |
57 | 22 | Marc Mengel | |
58 | 24 | Marc Mengel | * *ifdh* *leaveState* _state_ |
59 | 22 | Marc Mengel | |
60 | 22 | Marc Mengel | |
61 | 24 | Marc Mengel | * *ifdh* *createDefinition* _name dims user group_ |
62 | 22 | Marc Mengel | |
63 | 22 | Marc Mengel | make a named dataset definition from a dimension string |
64 | 22 | Marc Mengel | |
65 | 24 | Marc Mengel | * *ifdh* *deleteDefinition* _name_ |
66 | 22 | Marc Mengel | |
67 | 22 | Marc Mengel | remove data set definition |
68 | 22 | Marc Mengel | |
69 | 24 | Marc Mengel | * *ifdh* *describeDefinition* _name_ |
70 | 22 | Marc Mengel | |
71 | 22 | Marc Mengel | describe a named dataset definition |
72 | 22 | Marc Mengel | |
73 | 24 | Marc Mengel | * *ifdh* *translateConstraints* _dims_ |
74 | 22 | Marc Mengel | |
75 | 22 | Marc Mengel | give file list for dimension string |
76 | 22 | Marc Mengel | |
77 | 24 | Marc Mengel | * *ifdh* *locateFile* _name_ |
78 | 22 | Marc Mengel | |
79 | 22 | Marc Mengel | locate a file |
80 | 22 | Marc Mengel | |
81 | 24 | Marc Mengel | * *ifdh* *getMetadata* _name_ |
82 | 22 | Marc Mengel | |
83 | 22 | Marc Mengel | get a files metadata |
84 | 22 | Marc Mengel | |
85 | 24 | Marc Mengel | * *ifdh* *dumpStation* _name what_ |
86 | 22 | Marc Mengel | |
87 | 22 | Marc Mengel | give a dump of a SAM station status |
88 | 22 | Marc Mengel | |
89 | 24 | Marc Mengel | * *ifdh* *startProject* _name station defname_or_id user group_ |
90 | 22 | Marc Mengel | |
91 | 22 | Marc Mengel | start a new file delivery project |
92 | 22 | Marc Mengel | |
93 | 24 | Marc Mengel | * *ifdh* *findProject* _name station_ |
94 | 22 | Marc Mengel | |
95 | 22 | Marc Mengel | find a started project |
96 | 22 | Marc Mengel | |
97 | 24 | Marc Mengel | * *ifdh* *establishProcess* _projecturi appname appversion location user appfamily description filelimit_ |
98 | 22 | Marc Mengel | |
99 | 22 | Marc Mengel | set yourself up as a file consumer process for a project |
100 | 22 | Marc Mengel | |
101 | 24 | Marc Mengel | * *ifdh* *getNextFile* _projecturi processid_ |
102 | 22 | Marc Mengel | |
103 | 22 | Marc Mengel | get the next file location from a project |
104 | 22 | Marc Mengel | |
105 | 24 | Marc Mengel | * *ifdh* *updateFileStatus* _projecturi processid filename status_ |
106 | 22 | Marc Mengel | |
107 | 22 | Marc Mengel | update the file status (use: transferred, skipped, or consumed) |
108 | 22 | Marc Mengel | |
109 | 24 | Marc Mengel | * *ifdh* *endProcess* _projecturi processid_ |
110 | 22 | Marc Mengel | |
111 | 22 | Marc Mengel | end the process |
112 | 22 | Marc Mengel | |
113 | 24 | Marc Mengel | * *ifdh* *dumpProject* _projecturi_ |
114 | 22 | Marc Mengel | |
115 | 22 | Marc Mengel | say what the sam station knows about your process |
116 | 22 | Marc Mengel | |
117 | 24 | Marc Mengel | * *ifdh* *setStatus* _projecturi processid status_ |
118 | 22 | Marc Mengel | |
119 | 22 | Marc Mengel | set process status |
120 | 22 | Marc Mengel | |
121 | 24 | Marc Mengel | * *ifdh* *endProject* _projecturi_ |
122 | 22 | Marc Mengel | |
123 | 22 | Marc Mengel | end the project |
124 | 22 | Marc Mengel | |
125 | 24 | Marc Mengel | * *ifdh* *cleanup* |
126 | 22 | Marc Mengel | |
127 | 22 | Marc Mengel | clean up any tmp file stuff |
128 | 22 | Marc Mengel | |
129 | 24 | Marc Mengel | * *ifdh* *renameOutput* _how_ |
130 | 22 | Marc Mengel | |
131 | 22 | Marc Mengel | give output files reported with addOutputFile a unique name |
132 | 22 | Marc Mengel | |
133 | 24 | Marc Mengel | * *ifdh* *mv* _args_ |
134 | 22 | Marc Mengel | |
135 | 22 | Marc Mengel | general file rename using mvn or srmcp |
136 | 22 | Marc Mengel | |
137 | 24 | Marc Mengel | * *ifdh* *ls* _loc recursion_depth force_ |
138 | 22 | Marc Mengel | |
139 | 22 | Marc Mengel | Get a list of directory contents, or check existence of files |
140 | 22 | Marc Mengel | |
141 | 24 | Marc Mengel | * *ifdh* *mkdir* _loc force_ |
142 | 22 | Marc Mengel | |
143 | 22 | Marc Mengel | make a directory (i.e. for file destination) |
144 | 22 | Marc Mengel | |
145 | 24 | Marc Mengel | * *ifdh* *rm* _loc force_ |
146 | 22 | Marc Mengel | |
147 | 22 | Marc Mengel | remove files |
148 | 26 | Dennis Box | |
149 | 26 | Dennis Box | * *ifdh* *rmdir* loc force |
150 | 26 | Dennis Box | |
151 | 26 | Dennis Box | remove directories |
152 | 26 | Dennis Box | |
153 | 26 | Dennis Box | * *ifdh* *more* loc |
154 | 26 | Dennis Box | |
155 | 26 | Dennis Box | view text files |
156 | 26 | Dennis Box | |
157 | 26 | Dennis Box | * *ifdh* *pin* loc seconds |
158 | 26 | Dennis Box | |
159 | 26 | Dennis Box | pin file |
160 | 26 | Dennis Box | |
161 | 26 | Dennis Box | * *ifdh* *chmod* mode loc force |
162 | 26 | Dennis Box | |
163 | 26 | Dennis Box | change file permissions |
164 | 26 | Dennis Box | |
165 | 26 | Dennis Box | * *ifdh* *rename* loc loc2 force |
166 | 26 | Dennis Box | |
167 | 26 | Dennis Box | atomic rename items in same directory/fs |
168 | 26 | Dennis Box | |
169 | 26 | Dennis Box | * *ifdh* *ll* loc recursion_depth force |
170 | 26 | Dennis Box | |
171 | 26 | Dennis Box | list files with long listing |
172 | 26 | Dennis Box | use recursion_depth== 0 to check directory without contents |
173 | 26 | Dennis Box | |
174 | 26 | Dennis Box | * *ifdh* *lss* loc recursion_depth force |
175 | 26 | Dennis Box | |
176 | 26 | Dennis Box | list files with sizes |
177 | 26 | Dennis Box | use recursion_depth== 0 to check directory without contents |
178 | 26 | Dennis Box | |
179 | 26 | Dennis Box | * *ifdh* *findMatchingFiles* path glob |
180 | 26 | Dennis Box | |
181 | 26 | Dennis Box | find filenames and sizes matching pattern |
182 | 26 | Dennis Box | |
183 | 26 | Dennis Box | * *ifdh* *fetchSharedFiles* args |
184 | 26 | Dennis Box | |
185 | 26 | Dennis Box | filenames and sizes matching pattern moved locally enough to be seen |
186 | 26 | Dennis Box | |
187 | 26 | Dennis Box | * *ifdh* *locateFiles* args |
188 | 26 | Dennis Box | |
189 | 26 | Dennis Box | locate multiple files |
190 | 26 | Dennis Box | |
191 | 26 | Dennis Box | * *ifdh* *checksum* loc |
192 | 26 | Dennis Box | |
193 | 26 | Dennis Box | cheksum file |