Project

General

Profile

Ifdh commands » History » Version 30

Michael Kirby, 03/20/2017 10:15 AM

1 2 Marc Mengel
h1. ifdh commands
2 1 Marc Mengel
3 28 Marc Mengel
Note that many of these commands are new in ifdh v1_3_x and/or v1_7_x.
4 1 Marc Mengel
5 28 Marc Mengel
Many commands implicitly use a [[State Directory]].
6 28 Marc Mengel
7 1 Marc Mengel
h2. Environment variables
8 15 Marc Mengel
9 28 Marc Mengel
See [[Environment Variable Reference]]
10 15 Marc Mengel
11 15 Marc Mengel
h2. Commands
12 9 Marc Mengel
13 29 Marc Mengel
* *ifdh* *cp* _args_
14 22 Marc Mengel
	   
15 22 Marc Mengel
	 general file copy using cpn locks dd, gridftp, or srmcp 
16 22 Marc Mengel
	 supports:  
17 22 Marc Mengel
	   
18 23 Marc Mengel
	 * @cp src1 dest1 [';' src2 dest2 [';'...]]                     -- basic source/dest filenames @
19 23 Marc Mengel
	 * @cp -r src1 dest1 [';' src2 dest2 [';'...]]                  -- recursive directory copies @
20 23 Marc Mengel
	 * @cp -D src1 src2 destdir1 [';' src3 src4 destdir2 [';'...]]  -- copies to dest. directory @
21 23 Marc Mengel
	 * @cp -f file_with_src_space_dest_lines                        -- copies to a list file @
22 22 Marc Mengel
	 * any of the above can take --force={cpn,gridftp,srmcp,expgridftp} 
23 22 Marc Mengel
	 * any of the file/dest arguments can be URIs 
24 23 Marc Mengel
	 
25 1 Marc Mengel
26 30 Michael Kirby
	 Users might consider utilizing these environment variables to control the response of "ifdh cp" commands:
27 30 Michael Kirby
28 30 Michael Kirby
	 * @export IFDH_GRIDFTP_EXTRA="-st N"@ which sets the stall timeout at N seconds
29 30 Michael Kirby
	 * @export IFDH_CP_MAXRETRIES=N@, we find 2 is usually a good number
30 30 Michael Kirby
	 * note that IFDH_GRIDFTP_EXTRA can be used to pass many arguments to globus-url-copy http://toolkit.globus.org/toolkit/docs/3.2/gridftp/user/globusurlcopy.html
31 30 Michael Kirby
	 * full environment variables here: [[Environment Variable Reference]]
32 30 Michael Kirby
33 25 Marc Mengel
* *ifdh* *fetchInput* _src_uri_
34 22 Marc Mengel
	   
35 22 Marc Mengel
	 get input file to local scratch, return scratch location
36 22 Marc Mengel
37 25 Marc Mengel
* *ifdh* *localPath* _src_uri_
38 22 Marc Mengel
	   
39 22 Marc Mengel
	 return scratch location fetchInput would give, without copying
40 22 Marc Mengel
41 24 Marc Mengel
* *ifdh* *addOutputFile* _filename_
42 22 Marc Mengel
	   
43 22 Marc Mengel
	 add output file to set
44 22 Marc Mengel
45 25 Marc Mengel
* *ifdh* *copyBackOutput* _dest_dir_
46 22 Marc Mengel
	   
47 22 Marc Mengel
	 copy output file set to destination with cpn or srmcp
48 22 Marc Mengel
49 24 Marc Mengel
* *ifdh* *log* _args_
50 22 Marc Mengel
	   
51 22 Marc Mengel
	 logging 
52 22 Marc Mengel
53 25 Marc Mengel
* *ifdh* *enterState* _state_
54 22 Marc Mengel
	   
55 22 Marc Mengel
	 log entering/leaving states
56 22 Marc Mengel
57 24 Marc Mengel
* *ifdh* *leaveState* _state_
58 22 Marc Mengel
	  
59 22 Marc Mengel
60 24 Marc Mengel
* *ifdh* *createDefinition* _name  dims  user  group_
61 22 Marc Mengel
	   
62 22 Marc Mengel
	 make a named dataset definition from a dimension string
63 22 Marc Mengel
64 24 Marc Mengel
* *ifdh* *deleteDefinition* _name_
65 22 Marc Mengel
	   
66 22 Marc Mengel
	 remove data set definition
67 22 Marc Mengel
68 24 Marc Mengel
* *ifdh* *describeDefinition* _name_
69 22 Marc Mengel
	   
70 22 Marc Mengel
	 describe a named dataset definition
71 22 Marc Mengel
72 24 Marc Mengel
* *ifdh* *translateConstraints* _dims_
73 22 Marc Mengel
	   
74 22 Marc Mengel
	 give file list for dimension string
75 22 Marc Mengel
76 24 Marc Mengel
* *ifdh* *locateFile* _name_
77 22 Marc Mengel
	   
78 22 Marc Mengel
	 locate a file
79 22 Marc Mengel
80 24 Marc Mengel
* *ifdh* *getMetadata* _name_
81 22 Marc Mengel
	   
82 22 Marc Mengel
	 get a files metadata
83 22 Marc Mengel
84 24 Marc Mengel
* *ifdh* *dumpStation* _name  what_
85 22 Marc Mengel
	   
86 22 Marc Mengel
	 give a dump of a SAM station status
87 22 Marc Mengel
88 24 Marc Mengel
* *ifdh* *startProject* _name  station  defname_or_id  user  group_
89 22 Marc Mengel
	   
90 22 Marc Mengel
	 start a new file delivery project
91 22 Marc Mengel
92 24 Marc Mengel
* *ifdh* *findProject* _name  station_
93 22 Marc Mengel
	   
94 22 Marc Mengel
	 find a started project
95 22 Marc Mengel
96 24 Marc Mengel
* *ifdh* *establishProcess* _projecturi  appname  appversion  location  user  appfamily   description   filelimit_
97 22 Marc Mengel
	   
98 22 Marc Mengel
	 set yourself up as a file consumer process for a project
99 22 Marc Mengel
100 24 Marc Mengel
* *ifdh* *getNextFile* _projecturi  processid_
101 22 Marc Mengel
	   
102 22 Marc Mengel
	 get the next file location from a project
103 22 Marc Mengel
104 24 Marc Mengel
* *ifdh* *updateFileStatus* _projecturi  processid  filename  status_
105 22 Marc Mengel
	   
106 22 Marc Mengel
	 update the file status (use: transferred, skipped, or consumed)
107 22 Marc Mengel
108 24 Marc Mengel
* *ifdh* *endProcess* _projecturi  processid_
109 22 Marc Mengel
	   
110 22 Marc Mengel
	 end the process
111 22 Marc Mengel
112 24 Marc Mengel
* *ifdh* *dumpProject* _projecturi_
113 22 Marc Mengel
	   
114 22 Marc Mengel
	 say what the sam station knows about your process
115 22 Marc Mengel
116 24 Marc Mengel
* *ifdh* *setStatus* _projecturi  processid  status_
117 22 Marc Mengel
	   
118 22 Marc Mengel
	 set process status
119 22 Marc Mengel
120 24 Marc Mengel
* *ifdh* *endProject* _projecturi_
121 22 Marc Mengel
	   
122 22 Marc Mengel
	 end the project
123 22 Marc Mengel
124 24 Marc Mengel
* *ifdh* *cleanup* 
125 22 Marc Mengel
	   
126 22 Marc Mengel
	 clean up any tmp file stuff
127 22 Marc Mengel
128 24 Marc Mengel
* *ifdh* *renameOutput* _how_
129 22 Marc Mengel
	   
130 22 Marc Mengel
	 give output files reported with addOutputFile a unique name
131 22 Marc Mengel
132 24 Marc Mengel
* *ifdh* *mv* _args_
133 22 Marc Mengel
	   
134 22 Marc Mengel
	 general file rename using mvn or srmcp
135 22 Marc Mengel
136 24 Marc Mengel
* *ifdh* *ls* _loc  recursion_depth  force_
137 22 Marc Mengel
	   
138 22 Marc Mengel
	 Get a list of directory contents, or check existence of files
139 22 Marc Mengel
140 24 Marc Mengel
* *ifdh* *mkdir* _loc  force_
141 22 Marc Mengel
	   
142 22 Marc Mengel
	 make a directory (i.e. for file destination)
143 22 Marc Mengel
144 24 Marc Mengel
* *ifdh* *rm* _loc  force_
145 22 Marc Mengel
	   
146 22 Marc Mengel
	 remove files
147 26 Dennis Box
148 26 Dennis Box
* *ifdh* *rmdir* loc force
149 26 Dennis Box
150 26 Dennis Box
         remove directories
151 26 Dennis Box
	
152 26 Dennis Box
*  *ifdh* *more*  loc 
153 26 Dennis Box
	   
154 26 Dennis Box
	 view text files
155 26 Dennis Box
156 26 Dennis Box
* *ifdh* *pin*  loc  seconds 
157 26 Dennis Box
	   
158 26 Dennis Box
	 pin file
159 26 Dennis Box
160 26 Dennis Box
*  *ifdh* *chmod*  mode  loc  force  
161 26 Dennis Box
	   
162 26 Dennis Box
	 change file permissions
163 26 Dennis Box
164 26 Dennis Box
* *ifdh* *rename*  loc  loc2  force  
165 26 Dennis Box
	   
166 26 Dennis Box
	 atomic rename items in same directory/fs
167 26 Dennis Box
168 26 Dennis Box
*  *ifdh* *ll*  loc  recursion_depth  force  
169 26 Dennis Box
	   
170 26 Dennis Box
	 list files with long listing 
171 26 Dennis Box
	 use recursion_depth== 0 to check directory without contents
172 26 Dennis Box
173 26 Dennis Box
*  *ifdh* *lss*  loc  recursion_depth  force  
174 26 Dennis Box
	   
175 26 Dennis Box
	 list files with sizes 
176 26 Dennis Box
	 use recursion_depth== 0 to check directory without contents
177 26 Dennis Box
178 26 Dennis Box
*	*ifdh* *findMatchingFiles*  path  glob 
179 26 Dennis Box
	   
180 26 Dennis Box
	 find filenames and sizes matching pattern
181 26 Dennis Box
182 26 Dennis Box
*	*ifdh* *fetchSharedFiles* args
183 26 Dennis Box
	   
184 26 Dennis Box
	 filenames and sizes matching pattern moved locally enough to be seen
185 26 Dennis Box
186 26 Dennis Box
*	*ifdh* *locateFiles* args
187 26 Dennis Box
	   
188 26 Dennis Box
	 locate multiple files
189 26 Dennis Box
190 26 Dennis Box
*	*ifdh* *checksum*  loc 
191 26 Dennis Box
	   
192 26 Dennis Box
	 cheksum file