Project

General

Profile

Ifdh commands » History » Version 26

Dennis Box, 02/20/2015 05:58 PM

1 2 Marc Mengel
h1. ifdh commands
2 1 Marc Mengel
3 15 Marc Mengel
Note that many of these commands are new in ifdh v1_3_x.
4 15 Marc Mengel
5 15 Marc Mengel
h2. Environment variables
6 15 Marc Mengel
7 15 Marc Mengel
* IFDH_DEBUG=1 for voluminous debug output
8 15 Marc Mengel
* EXPERIMENT set to an experiment name ("minerva", "nova", etc.)
9 15 Marc Mengel
  if it can't be guessed from your group,etc.
10 15 Marc Mengel
* IFDH_BASE_URI to pick a particular samweb server.
11 15 Marc Mengel
* IFDH_GRIDFTP_EXTRA, IFDH_SRM_EXTRA, IFDH_DD_EXTRA flags to set options
12 15 Marc Mengel
  flor globus-url-copy, lcg_cp, dd, etc.
13 15 Marc Mengel
* IFDH_WEB_TIMEOUT for samweb timeouts.
14 15 Marc Mengel
15 15 Marc Mengel
h2. Commands
16 9 Marc Mengel
17 22 Marc Mengel
* *ifdh* *cp* _args_
18 22 Marc Mengel
	   
19 22 Marc Mengel
	 general file copy using cpn locks dd, gridftp, or srmcp 
20 22 Marc Mengel
	 supports:  
21 22 Marc Mengel
	   
22 23 Marc Mengel
	 * @cp src1 dest1 [';' src2 dest2 [';'...]]                     -- basic source/dest filenames @
23 23 Marc Mengel
	 * @cp -r src1 dest1 [';' src2 dest2 [';'...]]                  -- recursive directory copies @
24 23 Marc Mengel
	 * @cp -D src1 src2 destdir1 [';' src3 src4 destdir2 [';'...]]  -- copies to dest. directory @
25 23 Marc Mengel
	 * @cp -f file_with_src_space_dest_lines                        -- copies to a list file @
26 22 Marc Mengel
	 * any of the above can take --force={cpn,gridftp,srmcp,expgridftp} 
27 22 Marc Mengel
	 * any of the file/dest arguments can be URIs 
28 23 Marc Mengel
	 
29 1 Marc Mengel
30 25 Marc Mengel
* *ifdh* *fetchInput* _src_uri_
31 22 Marc Mengel
	   
32 22 Marc Mengel
	 get input file to local scratch, return scratch location
33 22 Marc Mengel
34 25 Marc Mengel
* *ifdh* *localPath* _src_uri_
35 22 Marc Mengel
	   
36 22 Marc Mengel
	 return scratch location fetchInput would give, without copying
37 22 Marc Mengel
38 24 Marc Mengel
* *ifdh* *addOutputFile* _filename_
39 22 Marc Mengel
	   
40 22 Marc Mengel
	 add output file to set
41 22 Marc Mengel
42 25 Marc Mengel
* *ifdh* *copyBackOutput* _dest_dir_
43 22 Marc Mengel
	   
44 22 Marc Mengel
	 copy output file set to destination with cpn or srmcp
45 22 Marc Mengel
46 24 Marc Mengel
* *ifdh* *log* _args_
47 22 Marc Mengel
	   
48 22 Marc Mengel
	 logging 
49 22 Marc Mengel
50 25 Marc Mengel
* *ifdh* *enterState* _state_
51 22 Marc Mengel
	   
52 22 Marc Mengel
	 log entering/leaving states
53 22 Marc Mengel
54 24 Marc Mengel
* *ifdh* *leaveState* _state_
55 22 Marc Mengel
	  
56 22 Marc Mengel
57 24 Marc Mengel
* *ifdh* *createDefinition* _name  dims  user  group_
58 22 Marc Mengel
	   
59 22 Marc Mengel
	 make a named dataset definition from a dimension string
60 22 Marc Mengel
61 24 Marc Mengel
* *ifdh* *deleteDefinition* _name_
62 22 Marc Mengel
	   
63 22 Marc Mengel
	 remove data set definition
64 22 Marc Mengel
65 24 Marc Mengel
* *ifdh* *describeDefinition* _name_
66 22 Marc Mengel
	   
67 22 Marc Mengel
	 describe a named dataset definition
68 22 Marc Mengel
69 24 Marc Mengel
* *ifdh* *translateConstraints* _dims_
70 22 Marc Mengel
	   
71 22 Marc Mengel
	 give file list for dimension string
72 22 Marc Mengel
73 24 Marc Mengel
* *ifdh* *locateFile* _name_
74 22 Marc Mengel
	   
75 22 Marc Mengel
	 locate a file
76 22 Marc Mengel
77 24 Marc Mengel
* *ifdh* *getMetadata* _name_
78 22 Marc Mengel
	   
79 22 Marc Mengel
	 get a files metadata
80 22 Marc Mengel
81 24 Marc Mengel
* *ifdh* *dumpStation* _name  what_
82 22 Marc Mengel
	   
83 22 Marc Mengel
	 give a dump of a SAM station status
84 22 Marc Mengel
85 24 Marc Mengel
* *ifdh* *startProject* _name  station  defname_or_id  user  group_
86 22 Marc Mengel
	   
87 22 Marc Mengel
	 start a new file delivery project
88 22 Marc Mengel
89 24 Marc Mengel
* *ifdh* *findProject* _name  station_
90 22 Marc Mengel
	   
91 22 Marc Mengel
	 find a started project
92 22 Marc Mengel
93 24 Marc Mengel
* *ifdh* *establishProcess* _projecturi  appname  appversion  location  user  appfamily   description   filelimit_
94 22 Marc Mengel
	   
95 22 Marc Mengel
	 set yourself up as a file consumer process for a project
96 22 Marc Mengel
97 24 Marc Mengel
* *ifdh* *getNextFile* _projecturi  processid_
98 22 Marc Mengel
	   
99 22 Marc Mengel
	 get the next file location from a project
100 22 Marc Mengel
101 24 Marc Mengel
* *ifdh* *updateFileStatus* _projecturi  processid  filename  status_
102 22 Marc Mengel
	   
103 22 Marc Mengel
	 update the file status (use: transferred, skipped, or consumed)
104 22 Marc Mengel
105 24 Marc Mengel
* *ifdh* *endProcess* _projecturi  processid_
106 22 Marc Mengel
	   
107 22 Marc Mengel
	 end the process
108 22 Marc Mengel
109 24 Marc Mengel
* *ifdh* *dumpProject* _projecturi_
110 22 Marc Mengel
	   
111 22 Marc Mengel
	 say what the sam station knows about your process
112 22 Marc Mengel
113 24 Marc Mengel
* *ifdh* *setStatus* _projecturi  processid  status_
114 22 Marc Mengel
	   
115 22 Marc Mengel
	 set process status
116 22 Marc Mengel
117 24 Marc Mengel
* *ifdh* *endProject* _projecturi_
118 22 Marc Mengel
	   
119 22 Marc Mengel
	 end the project
120 22 Marc Mengel
121 24 Marc Mengel
* *ifdh* *cleanup* 
122 22 Marc Mengel
	   
123 22 Marc Mengel
	 clean up any tmp file stuff
124 22 Marc Mengel
125 24 Marc Mengel
* *ifdh* *renameOutput* _how_
126 22 Marc Mengel
	   
127 22 Marc Mengel
	 give output files reported with addOutputFile a unique name
128 22 Marc Mengel
129 24 Marc Mengel
* *ifdh* *mv* _args_
130 22 Marc Mengel
	   
131 22 Marc Mengel
	 general file rename using mvn or srmcp
132 22 Marc Mengel
133 24 Marc Mengel
* *ifdh* *ls* _loc  recursion_depth  force_
134 22 Marc Mengel
	   
135 22 Marc Mengel
	 Get a list of directory contents, or check existence of files
136 22 Marc Mengel
137 24 Marc Mengel
* *ifdh* *mkdir* _loc  force_
138 22 Marc Mengel
	   
139 22 Marc Mengel
	 make a directory (i.e. for file destination)
140 22 Marc Mengel
141 24 Marc Mengel
* *ifdh* *rm* _loc  force_
142 22 Marc Mengel
	   
143 22 Marc Mengel
	 remove files
144 26 Dennis Box
145 26 Dennis Box
* *ifdh* *rmdir* loc force
146 26 Dennis Box
147 26 Dennis Box
         remove directories
148 26 Dennis Box
	
149 26 Dennis Box
*  *ifdh* *more*  loc 
150 26 Dennis Box
	   
151 26 Dennis Box
	 view text files
152 26 Dennis Box
153 26 Dennis Box
* *ifdh* *pin*  loc  seconds 
154 26 Dennis Box
	   
155 26 Dennis Box
	 pin file
156 26 Dennis Box
157 26 Dennis Box
*  *ifdh* *chmod*  mode  loc  force  
158 26 Dennis Box
	   
159 26 Dennis Box
	 change file permissions
160 26 Dennis Box
161 26 Dennis Box
* *ifdh* *rename*  loc  loc2  force  
162 26 Dennis Box
	   
163 26 Dennis Box
	 atomic rename items in same directory/fs
164 26 Dennis Box
165 26 Dennis Box
*  *ifdh* *ll*  loc  recursion_depth  force  
166 26 Dennis Box
	   
167 26 Dennis Box
	 list files with long listing 
168 26 Dennis Box
	 use recursion_depth== 0 to check directory without contents
169 26 Dennis Box
170 26 Dennis Box
*  *ifdh* *lss*  loc  recursion_depth  force  
171 26 Dennis Box
	   
172 26 Dennis Box
	 list files with sizes 
173 26 Dennis Box
	 use recursion_depth== 0 to check directory without contents
174 26 Dennis Box
175 26 Dennis Box
*	*ifdh* *findMatchingFiles*  path  glob 
176 26 Dennis Box
	   
177 26 Dennis Box
	 find filenames and sizes matching pattern
178 26 Dennis Box
179 26 Dennis Box
*	*ifdh* *fetchSharedFiles* args
180 26 Dennis Box
	   
181 26 Dennis Box
	 filenames and sizes matching pattern moved locally enough to be seen
182 26 Dennis Box
183 26 Dennis Box
*	*ifdh* *locateFiles* args
184 26 Dennis Box
	   
185 26 Dennis Box
	 locate multiple files
186 26 Dennis Box
187 26 Dennis Box
*	*ifdh* *checksum*  loc 
188 26 Dennis Box
	   
189 26 Dennis Box
	 cheksum file