Project

General

Profile

Ifdh commands » History » Version 32

Marc Mengel, 06/09/2017 10:20 AM

1 2 Marc Mengel
h1. ifdh commands
2 1 Marc Mengel
3 28 Marc Mengel
Note that many of these commands are new in ifdh v1_3_x and/or v1_7_x.
4 1 Marc Mengel
5 28 Marc Mengel
Many commands implicitly use a [[State Directory]].
6 28 Marc Mengel
7 1 Marc Mengel
h2. Environment variables
8 15 Marc Mengel
9 28 Marc Mengel
See [[Environment Variable Reference]]
10 15 Marc Mengel
11 15 Marc Mengel
h2. Commands
12 9 Marc Mengel
13 29 Marc Mengel
* *ifdh* *cp* _args_
14 22 Marc Mengel
	   
15 22 Marc Mengel
	 general file copy using cpn locks dd, gridftp, or srmcp 
16 22 Marc Mengel
	 supports:  
17 22 Marc Mengel
	   
18 23 Marc Mengel
	 * @cp src1 dest1 [';' src2 dest2 [';'...]]                     -- basic source/dest filenames @
19 23 Marc Mengel
	 * @cp -r src1 dest1 [';' src2 dest2 [';'...]]                  -- recursive directory copies @
20 23 Marc Mengel
	 * @cp -D src1 src2 destdir1 [';' src3 src4 destdir2 [';'...]]  -- copies to dest. directory @
21 23 Marc Mengel
	 * @cp -f file_with_src_space_dest_lines                        -- copies to a list file @
22 32 Marc Mengel
	 * any of the above can take --force={cpn,gridftp,srmcp,rootd,http} 
23 22 Marc Mengel
	 * any of the file/dest arguments can be URIs 
24 23 Marc Mengel
	 
25 1 Marc Mengel
26 30 Michael Kirby
	 Users might consider utilizing these environment variables to control the response of "ifdh cp" commands:
27 30 Michael Kirby
28 31 Michael Kirby
	 * @export IFDH_GRIDFTP_EXTRA="-st N"@ which sets the stall timeout at N seconds IFDH_XROOTD_EXTRA
29 30 Michael Kirby
	 * @export IFDH_CP_MAXRETRIES=N@, we find 2 is usually a good number
30 1 Marc Mengel
	 * note that IFDH_GRIDFTP_EXTRA can be used to pass many arguments to globus-url-copy http://toolkit.globus.org/toolkit/docs/3.2/gridftp/user/globusurlcopy.html
31 31 Michael Kirby
	 * note that IFDH_XROOTD_EXTRA can be used to pass many arguments to xrootd/xrdcp actions https://linux.die.net/man/1/xrdcp 
32 30 Michael Kirby
	 * full environment variables here: [[Environment Variable Reference]]
33 30 Michael Kirby
34 25 Marc Mengel
* *ifdh* *fetchInput* _src_uri_
35 22 Marc Mengel
	   
36 22 Marc Mengel
	 get input file to local scratch, return scratch location
37 22 Marc Mengel
38 25 Marc Mengel
* *ifdh* *localPath* _src_uri_
39 22 Marc Mengel
	   
40 22 Marc Mengel
	 return scratch location fetchInput would give, without copying
41 22 Marc Mengel
42 24 Marc Mengel
* *ifdh* *addOutputFile* _filename_
43 22 Marc Mengel
	   
44 22 Marc Mengel
	 add output file to set
45 22 Marc Mengel
46 25 Marc Mengel
* *ifdh* *copyBackOutput* _dest_dir_
47 22 Marc Mengel
	   
48 22 Marc Mengel
	 copy output file set to destination with cpn or srmcp
49 22 Marc Mengel
50 24 Marc Mengel
* *ifdh* *log* _args_
51 22 Marc Mengel
	   
52 22 Marc Mengel
	 logging 
53 22 Marc Mengel
54 25 Marc Mengel
* *ifdh* *enterState* _state_
55 22 Marc Mengel
	   
56 22 Marc Mengel
	 log entering/leaving states
57 22 Marc Mengel
58 24 Marc Mengel
* *ifdh* *leaveState* _state_
59 22 Marc Mengel
	  
60 22 Marc Mengel
61 24 Marc Mengel
* *ifdh* *createDefinition* _name  dims  user  group_
62 22 Marc Mengel
	   
63 22 Marc Mengel
	 make a named dataset definition from a dimension string
64 22 Marc Mengel
65 24 Marc Mengel
* *ifdh* *deleteDefinition* _name_
66 22 Marc Mengel
	   
67 22 Marc Mengel
	 remove data set definition
68 22 Marc Mengel
69 24 Marc Mengel
* *ifdh* *describeDefinition* _name_
70 22 Marc Mengel
	   
71 22 Marc Mengel
	 describe a named dataset definition
72 22 Marc Mengel
73 24 Marc Mengel
* *ifdh* *translateConstraints* _dims_
74 22 Marc Mengel
	   
75 22 Marc Mengel
	 give file list for dimension string
76 22 Marc Mengel
77 24 Marc Mengel
* *ifdh* *locateFile* _name_
78 22 Marc Mengel
	   
79 22 Marc Mengel
	 locate a file
80 22 Marc Mengel
81 24 Marc Mengel
* *ifdh* *getMetadata* _name_
82 22 Marc Mengel
	   
83 22 Marc Mengel
	 get a files metadata
84 22 Marc Mengel
85 24 Marc Mengel
* *ifdh* *dumpStation* _name  what_
86 22 Marc Mengel
	   
87 22 Marc Mengel
	 give a dump of a SAM station status
88 22 Marc Mengel
89 24 Marc Mengel
* *ifdh* *startProject* _name  station  defname_or_id  user  group_
90 22 Marc Mengel
	   
91 22 Marc Mengel
	 start a new file delivery project
92 22 Marc Mengel
93 24 Marc Mengel
* *ifdh* *findProject* _name  station_
94 22 Marc Mengel
	   
95 22 Marc Mengel
	 find a started project
96 22 Marc Mengel
97 24 Marc Mengel
* *ifdh* *establishProcess* _projecturi  appname  appversion  location  user  appfamily   description   filelimit_
98 22 Marc Mengel
	   
99 22 Marc Mengel
	 set yourself up as a file consumer process for a project
100 22 Marc Mengel
101 24 Marc Mengel
* *ifdh* *getNextFile* _projecturi  processid_
102 22 Marc Mengel
	   
103 22 Marc Mengel
	 get the next file location from a project
104 22 Marc Mengel
105 24 Marc Mengel
* *ifdh* *updateFileStatus* _projecturi  processid  filename  status_
106 22 Marc Mengel
	   
107 22 Marc Mengel
	 update the file status (use: transferred, skipped, or consumed)
108 22 Marc Mengel
109 24 Marc Mengel
* *ifdh* *endProcess* _projecturi  processid_
110 22 Marc Mengel
	   
111 22 Marc Mengel
	 end the process
112 22 Marc Mengel
113 24 Marc Mengel
* *ifdh* *dumpProject* _projecturi_
114 22 Marc Mengel
	   
115 22 Marc Mengel
	 say what the sam station knows about your process
116 22 Marc Mengel
117 24 Marc Mengel
* *ifdh* *setStatus* _projecturi  processid  status_
118 22 Marc Mengel
	   
119 22 Marc Mengel
	 set process status
120 22 Marc Mengel
121 24 Marc Mengel
* *ifdh* *endProject* _projecturi_
122 22 Marc Mengel
	   
123 22 Marc Mengel
	 end the project
124 22 Marc Mengel
125 24 Marc Mengel
* *ifdh* *cleanup* 
126 22 Marc Mengel
	   
127 22 Marc Mengel
	 clean up any tmp file stuff
128 22 Marc Mengel
129 24 Marc Mengel
* *ifdh* *renameOutput* _how_
130 22 Marc Mengel
	   
131 22 Marc Mengel
	 give output files reported with addOutputFile a unique name
132 22 Marc Mengel
133 24 Marc Mengel
* *ifdh* *mv* _args_
134 22 Marc Mengel
	   
135 22 Marc Mengel
	 general file rename using mvn or srmcp
136 22 Marc Mengel
137 24 Marc Mengel
* *ifdh* *ls* _loc  recursion_depth  force_
138 22 Marc Mengel
	   
139 22 Marc Mengel
	 Get a list of directory contents, or check existence of files
140 22 Marc Mengel
141 24 Marc Mengel
* *ifdh* *mkdir* _loc  force_
142 22 Marc Mengel
	   
143 22 Marc Mengel
	 make a directory (i.e. for file destination)
144 22 Marc Mengel
145 24 Marc Mengel
* *ifdh* *rm* _loc  force_
146 22 Marc Mengel
	   
147 22 Marc Mengel
	 remove files
148 26 Dennis Box
149 26 Dennis Box
* *ifdh* *rmdir* loc force
150 26 Dennis Box
151 26 Dennis Box
         remove directories
152 26 Dennis Box
	
153 26 Dennis Box
*  *ifdh* *more*  loc 
154 26 Dennis Box
	   
155 26 Dennis Box
	 view text files
156 26 Dennis Box
157 26 Dennis Box
* *ifdh* *pin*  loc  seconds 
158 26 Dennis Box
	   
159 26 Dennis Box
	 pin file
160 26 Dennis Box
161 26 Dennis Box
*  *ifdh* *chmod*  mode  loc  force  
162 26 Dennis Box
	   
163 26 Dennis Box
	 change file permissions
164 26 Dennis Box
165 26 Dennis Box
* *ifdh* *rename*  loc  loc2  force  
166 26 Dennis Box
	   
167 26 Dennis Box
	 atomic rename items in same directory/fs
168 26 Dennis Box
169 26 Dennis Box
*  *ifdh* *ll*  loc  recursion_depth  force  
170 26 Dennis Box
	   
171 26 Dennis Box
	 list files with long listing 
172 26 Dennis Box
	 use recursion_depth== 0 to check directory without contents
173 26 Dennis Box
174 26 Dennis Box
*  *ifdh* *lss*  loc  recursion_depth  force  
175 26 Dennis Box
	   
176 26 Dennis Box
	 list files with sizes 
177 26 Dennis Box
	 use recursion_depth== 0 to check directory without contents
178 26 Dennis Box
179 26 Dennis Box
*	*ifdh* *findMatchingFiles*  path  glob 
180 26 Dennis Box
	   
181 26 Dennis Box
	 find filenames and sizes matching pattern
182 26 Dennis Box
183 26 Dennis Box
*	*ifdh* *fetchSharedFiles* args
184 26 Dennis Box
	   
185 26 Dennis Box
	 filenames and sizes matching pattern moved locally enough to be seen
186 26 Dennis Box
187 26 Dennis Box
*	*ifdh* *locateFiles* args
188 26 Dennis Box
	   
189 26 Dennis Box
	 locate multiple files
190 26 Dennis Box
191 26 Dennis Box
*	*ifdh* *checksum*  loc 
192 26 Dennis Box
	   
193 26 Dennis Box
	 cheksum file