Project

General

Profile

SAM web cookbook » History » Version 1

Gavin Davies, 04/24/2014 08:03 PM

1 1 Gavin Davies
h1. SAM Web Cookbook (Nova Edition)
2 1 Gavin Davies
3 1 Gavin Davies
Right now SAM web client does not get setup by default so one must set it up themselves (this will change in due course!)
4 1 Gavin Davies
Setup SAM Web Client:
5 1 Gavin Davies
<pre>
6 1 Gavin Davies
export PRODUCTS=/grid/fermiapp/products/common/db/:$PRODUCTS
7 1 Gavin Davies
setup sam_web_client <version>
8 1 Gavin Davies
</pre>
9 1 Gavin Davies
Version is optional, use it if you want a specific version of the client (instead of the latest production one)
10 1 Gavin Davies
11 1 Gavin Davies
Set your experiment with an environment variable (this saves you typing later):
12 1 Gavin Davies
<pre>
13 1 Gavin Davies
export EXPERIMENT=nova
14 1 Gavin Davies
</pre>
15 1 Gavin Davies
16 1 Gavin Davies
If you don't have a certificate, get one (based off your kerberos credentials):
17 1 Gavin Davies
<pre>
18 1 Gavin Davies
kx509
19 1 Gavin Davies
</pre>
20 1 Gavin Davies
21 1 Gavin Davies
All of this can be automated by putting it in your login files (i.e. .bash_profile or .bashrc)
22 1 Gavin Davies
23 1 Gavin Davies
{{TOC}}
24 1 Gavin Davies
25 1 Gavin Davies
h1. Glossary and Conventions
26 1 Gavin Davies
27 1 Gavin Davies
The following shorthands are used:
28 1 Gavin Davies
29 1 Gavin Davies
In the following, anything with a set of angled brackets denotes a variable.  i.e. <run number> would be insert your own personal run number you were interested in.
30 1 Gavin Davies
31 1 Gavin Davies
Anything with a dollar sign in front of it denotes a shell variable, i.e. $BASE_QUERY
32 1 Gavin Davies
33 1 Gavin Davies
* BASE_QUERY is the data tier and detector. It is assumed to be set like:
34 1 Gavin Davies
35 1 Gavin Davies
<pre>
36 1 Gavin Davies
export BASE_QUERY="data_tier raw AND online.detector fardet"
37 1 Gavin Davies
</pre>
38 1 Gavin Davies
 
39 1 Gavin Davies
To get help from samweb type:
40 1 Gavin Davies
<pre>
41 1 Gavin Davies
samweb --help-commands
42 1 Gavin Davies
</pre>
43 1 Gavin Davies
44 1 Gavin Davies
h1. Beginner Recipes (boiling water)
45 1 Gavin Davies
46 1 Gavin Davies
<pre>
47 1 Gavin Davies
export BASE_QUERY="data_tier raw AND online.detector fardet"
48 1 Gavin Davies
</pre>
49 1 Gavin Davies
To save some typing.
50 1 Gavin Davies
51 1 Gavin Davies
Queries are NOT case SeNsItIvE
52 1 Gavin Davies
53 1 Gavin Davies
To save typing some parts of earlier queries are denoted as $QUERY_XXX
54 1 Gavin Davies
55 1 Gavin Davies
Any where you see a "list-files" you can replace it with a "count-files" to just return a count instead of actual file names.
56 1 Gavin Davies
57 1 Gavin Davies
58 1 Gavin Davies
h2. List Files from a data tier and detector
59 1 Gavin Davies
60 1 Gavin Davies
61 1 Gavin Davies
*samweb list-files "data_tier <tier> AND online.detector <det>"
62 1 Gavin Davies
63 1 Gavin Davies
Example:
64 1 Gavin Davies
<pre>
65 1 Gavin Davies
samweb list-files "data_tier raw AND online.detector fardet"
66 1 Gavin Davies
</pre>
67 1 Gavin Davies
68 1 Gavin Davies
Will return 459,532 files (today)
69 1 Gavin Davies
70 1 Gavin Davies
<pre>
71 1 Gavin Davies
samweb list-files "data_tier raw AND online.detector ndos"
72 1 Gavin Davies
</pre>
73 1 Gavin Davies
Will return 91,360 files (today)
74 1 Gavin Davies
75 1 Gavin Davies
From here on we use $BASE_QUERY for this.
76 1 Gavin Davies
77 1 Gavin Davies
h2. List Files from a Run
78 1 Gavin Davies
79 1 Gavin Davies
*samweb list-files "$BASE_QUERY and online.runumber <runnumber>"*
80 1 Gavin Davies
81 1 Gavin Davies
Or:
82 1 Gavin Davies
83 1 Gavin Davies
*samweb list-files "$BASE_QUERY and run_number <runnumber>"*
84 1 Gavin Davies
85 1 Gavin Davies
First one is DAQ specific, the other is more general.
86 1 Gavin Davies
87 1 Gavin Davies
Example:
88 1 Gavin Davies
<pre>
89 1 Gavin Davies
samweb list-files "$BASE_QUERY and run_number 13114"
90 1 Gavin Davies
</pre>
91 1 Gavin Davies
92 1 Gavin Davies
h2. List Files from a Time Period
93 1 Gavin Davies
94 1 Gavin Davies
Files created between two times:
95 1 Gavin Davies
96 1 Gavin Davies
*samweb list-files "$BASE_QUERY and start_time > '2014-01-30T23:29:00' and start_time < '2014-01-31T00:30:00'"*
97 1 Gavin Davies
98 1 Gavin Davies
Example:
99 1 Gavin Davies
<pre>
100 1 Gavin Davies
samweb list-files "$BASE_QUERY and start_time > '2014-01-30T23:29:00' and start_time < '2014-01-31T00:30:00'"
101 1 Gavin Davies
</pre>
102 1 Gavin Davies
103 1 Gavin Davies
104 1 Gavin Davies
h2. List Files from a specific trigger stream
105 1 Gavin Davies
106 1 Gavin Davies
You want only a given stream.
107 1 Gavin Davies
108 1 Gavin Davies
*samweb list-files "$BASE_QUERY and run_number <run_no> and data_stream <stream>"*
109 1 Gavin Davies
110 1 Gavin Davies
or
111 1 Gavin Davies
112 1 Gavin Davies
*samweb list-files "$BASE_QUERY and run_number <run_no> and Online.Stream <stream>"*
113 1 Gavin Davies
114 1 Gavin Davies
For DAQ files only.
115 1 Gavin Davies
116 1 Gavin Davies
Stream is a number.  Streams are fully configurable, but in general in early 2014 they looked like:
117 1 Gavin Davies
118 1 Gavin Davies
* 0 = NuMI
119 1 Gavin Davies
* 1 = Booster Beam
120 1 Gavin Davies
* 2 = Min Bias
121 1 Gavin Davies
122 1 Gavin Davies
<pre>
123 1 Gavin Davies
samweb list-files "$BASE_QUERY and run_number 13114 and data_stream 0
124 1 Gavin Davies
</pre>
125 1 Gavin Davies
126 1 Gavin Davies
h2. List Files from DAQ Partition
127 1 Gavin Davies
128 1 Gavin Davies
You want only a specific DAQ Partition
129 1 Gavin Davies
130 1 Gavin Davies
*samweb list-file "$BASE_QUERY and Online.Partition <partno>"*
131 1 Gavin Davies
132 1 Gavin Davies
<pre>
133 1 Gavin Davies
*samweb list-file "$BASE_QUERY and Online.Partition 1"*
134 1 Gavin Davies
</pre>
135 1 Gavin Davies
136 1 Gavin Davies
h2. List Metadata associated with a file:
137 1 Gavin Davies
138 1 Gavin Davies
File names do not have paths, just base names (all files in SAM are unique)
139 1 Gavin Davies
140 1 Gavin Davies
*samweb get-metadata <filename>*
141 1 Gavin Davies
142 1 Gavin Davies
<pre>
143 1 Gavin Davies
samweb get-metadata fardet_r00013114_s20_t00.raw
144 1 Gavin Davies
</pre>
145 1 Gavin Davies
146 1 Gavin Davies
You get output like:
147 1 Gavin Davies
<pre>
148 1 Gavin Davies
                    File Name: fardet_r00013114_s20_t00.raw
149 1 Gavin Davies
                      File Id: 4877797
150 1 Gavin Davies
                    File Type: importedDetector
151 1 Gavin Davies
                  File Format: raw
152 1 Gavin Davies
                    File Size: 6908296
153 1 Gavin Davies
                          Crc: 74650857 (adler 32 crc type)
154 1 Gavin Davies
               Content Status: good
155 1 Gavin Davies
                        Group: nova
156 1 Gavin Davies
                    Data Tier: raw
157 1 Gavin Davies
                  Application: online datalogger 33
158 1 Gavin Davies
                  Event Count: 110
159 1 Gavin Davies
                  First Event: 171026
160 1 Gavin Davies
                   Last Event: 179507
161 1 Gavin Davies
                   Start Time: 2014-02-14T01:34:14
162 1 Gavin Davies
                     End Time: 2014-02-14T01:37:43
163 1 Gavin Davies
                  Data Stream: 0
164 1 Gavin Davies
             Online.ConfigIDX: 0
165 1 Gavin Davies
          Online.DataLoggerID: 1
166 1 Gavin Davies
     Online.DataLoggerVersion: 33
167 1 Gavin Davies
              Online.Detector: fardet
168 1 Gavin Davies
            Online.DetectorID: 2
169 1 Gavin Davies
             Online.Partition: 1
170 1 Gavin Davies
          Online.RunControlID: 0
171 1 Gavin Davies
     Online.RunControlVersion: 0
172 1 Gavin Davies
            Online.RunEndTime: 1392341863
173 1 Gavin Davies
             Online.RunNumber: 13114
174 1 Gavin Davies
               Online.RunSize: 1727074
175 1 Gavin Davies
          Online.RunStartTime: 1392337488
176 1 Gavin Davies
               Online.RunType: 0
177 1 Gavin Davies
                Online.Stream: 0
178 1 Gavin Davies
         Online.SubRunEndTime: 1392341863
179 1 Gavin Davies
       Online.SubRunStartTime: 1392341654
180 1 Gavin Davies
                Online.Subrun: 20
181 1 Gavin Davies
           Online.TotalEvents: 110
182 1 Gavin Davies
         Online.TriggerCtrlID: 0
183 1 Gavin Davies
        Online.TriggerListIDX: 0
184 1 Gavin Davies
Online.TriggerPrescaleListIDX: 0
185 1 Gavin Davies
        Online.TriggerVersion: 0
186 1 Gavin Davies
 Online.ValidTriggerTypesHigh: 0
187 1 Gavin Davies
Online.ValidTriggerTypesHigh2: 0
188 1 Gavin Davies
  Online.ValidTriggerTypesLow: 0
189 1 Gavin Davies
                         Runs: 13114.0020 (online)
190 1 Gavin Davies
               File Partition: 20
191 1 Gavin Davies
</pre>
192 1 Gavin Davies
193 1 Gavin Davies
h2. List files with some other parameter or parameters
194 1 Gavin Davies
195 1 Gavin Davies
*samweb list-file "$BASE_QUERY and Parameter.name_1 <value> and Parameter.name_2 <value>"*
196 1 Gavin Davies
197 1 Gavin Davies
<pre>
198 1 Gavin Davies
*samweb list-file "$BASE_QUERY and Online.TotalEvents > 123 and Online.DataLoggerVersion = 33"*
199 1 Gavin Davies
</pre>
200 1 Gavin Davies
201 1 Gavin Davies
202 1 Gavin Davies
h2. Get File locations
203 1 Gavin Davies
204 1 Gavin Davies
samweb locate-file <filename>
205 1 Gavin Davies
206 1 Gavin Davies
<pre>
207 1 Gavin Davies
samweb locate-file ndos_r00015701_s07_cosmic.raw
208 1 Gavin Davies
</pre>
209 1 Gavin Davies
Response will be a list of locations:
210 1 Gavin Davies
<pre>
211 1 Gavin Davies
novadata:/nova/data/rawdata/NDOS/000157/15701/cosmic
212 1 Gavin Davies
enstore:/pnfs/nova/rawdata/NDOS/runs/000157/15701(1548@vpe048)
213 1 Gavin Davies
</pre>
214 1 Gavin Davies
215 1 Gavin Davies
* Locations starting with "novadata" are bluearc central disk.
216 1 Gavin Davies
* Locations starting with "enstore" are dCache/Enstore locations (disk cache, tape backed)
217 1 Gavin Davies
218 1 Gavin Davies
h2. Get Ancestors of a File
219 1 Gavin Davies
220 1 Gavin Davies
*samweb file-lineage <children/descendants> <filename>*
221 1 Gavin Davies
222 1 Gavin Davies
Children are files derived directly from the input file
223 1 Gavin Davies
<pre>
224 1 Gavin Davies
samweb file-lineage children fardet_r00013096_s14_t00.raw
225 1 Gavin Davies
fardet_r00013096_s14_t00_numi_S14-01-20_v1_data.daq.root
226 1 Gavin Davies
</pre>
227 1 Gavin Davies
228 1 Gavin Davies
*samweb file-lineage <parents/ancenstors> <filename>*
229 1 Gavin Davies
230 1 Gavin Davies
<pre>
231 1 Gavin Davies
samweb file-lineage parents fardet_r00013096_s14_t00_numi_S14-01-20_v1_data.daq.root
232 1 Gavin Davies
fardet_r00013096_s14_t00.raw
233 1 Gavin Davies
</pre>
234 1 Gavin Davies
235 1 Gavin Davies
236 1 Gavin Davies
237 1 Gavin Davies
h1. Intermediate Recipes (Poaching eggs)
238 1 Gavin Davies
239 1 Gavin Davies
240 1 Gavin Davies
h2. Get a list of all currently defined fields
241 1 Gavin Davies
242 1 Gavin Davies
Go to:
243 1 Gavin Davies
"Current Nova Experiment Dimensions":http://samweb.fnal.gov:8480/sam/nova/api/files/list/dimensions
244 1 Gavin Davies
245 1 Gavin Davies
h2. Get a list of Non-DAQ data files (e.g. Laser Scans) matching a search
246 1 Gavin Davies
247 1 Gavin Davies
samweb list-file "data_tier laser_scan AND laser_scan.block_number = 23 AND laser_scan.layer_number > 4"
248 1 Gavin Davies
249 1 Gavin Davies
h2. Listing Files with children matching a selection
250 1 Gavin Davies
251 1 Gavin Davies
List raw files who have been processed through a different stage
252 1 Gavin Davies
253 1 Gavin Davies
samweb list-file "$BASE_QUERY and isparentof: (data_tier <stage> AND Parameter.name_1 <value>)"
254 1 Gavin Davies
255 1 Gavin Davies
<pre>
256 1 Gavin Davies
samweb list-files "$BASE_QUERY and isparentof: ( data_tier artdaq AND daq2rawdigit.base_release 'S14-01-20' )"
257 1 Gavin Davies
</pre>
258 1 Gavin Davies
259 1 Gavin Davies
h2. Listing Files that match a filename patern
260 1 Gavin Davies
261 1 Gavin Davies
This is to match parts of the file name
262 1 Gavin Davies
263 1 Gavin Davies
<pre>
264 1 Gavin Davies
samweb list-file "file_name like fardet%DDenergy%"
265 1 Gavin Davies
266 1 Gavin Davies
</pre>
267 1 Gavin Davies
268 1 Gavin Davies
h2. Listing Files with parents matching a selection
269 1 Gavin Davies
270 1 Gavin Davies
With BASE_QUERY2="data_tier artdaq AND online.detector fardet"
271 1 Gavin Davies
272 1 Gavin Davies
<pre>
273 1 Gavin Davies
samweb list-file "$BASE_QUERY2 and ischildof: ( data_tier raw AND Online.Subrun < 20)
274 1 Gavin Davies
</pre>
275 1 Gavin Davies
276 1 Gavin Davies
h2. Listing Files with no physical locations
277 1 Gavin Davies
278 1 Gavin Davies
*samweb list-files "$BASE_QUERY AND availability: virtual"*
279 1 Gavin Davies
280 1 Gavin Davies
<pre>
281 1 Gavin Davies
samweb list-files "$BASE_QUERY AND availability: virtual"
282 1 Gavin Davies
</pre>
283 1 Gavin Davies
284 1 Gavin Davies
285 1 Gavin Davies
h2. Listing Files with physical locations
286 1 Gavin Davies
287 1 Gavin Davies
*samweb list-files "$BASE_QUERY AND availability: physical"*
288 1 Gavin Davies
289 1 Gavin Davies
<pre>
290 1 Gavin Davies
samweb list-files "$BASE_QUERY AND availability: physical"
291 1 Gavin Davies
</pre>
292 1 Gavin Davies
293 1 Gavin Davies
h2. Retrieving Files with a physical location
294 1 Gavin Davies
295 1 Gavin Davies
You can retrieve files either individually or with a query pattern (multiple files).
296 1 Gavin Davies
297 1 Gavin Davies
h3. Retrieve a single file
298 1 Gavin Davies
299 1 Gavin Davies
*ifdh_fetch <filename>*
300 1 Gavin Davies
301 1 Gavin Davies
<pre>
302 1 Gavin Davies
ifdh_fetch fardet_r00012006_s61_t02.raw
303 1 Gavin Davies
</pre>
304 1 Gavin Davies
305 1 Gavin Davies
Note: you must have a valid certificate (i.e. run kx509)
306 1 Gavin Davies
307 1 Gavin Davies
h3. Retrieve a group of files
308 1 Gavin Davies
309 1 Gavin Davies
*ifdh_fetch `ifdh translateContraints <dimensions string>`*
310 1 Gavin Davies
311 1 Gavin Davies
<pre>
312 1 Gavin Davies
ifdh_fetch `ifdh translateConstraints "data_tier raw AND online.detector fardet and run_number 12006.51"`
313 1 Gavin Davies
</pre>
314 1 Gavin Davies
315 1 Gavin Davies
Note: Here ifdh is used to do the lookup of the files and then the resulting names are passed to the fetch.
316 1 Gavin Davies
317 1 Gavin Davies
h2. Verifying that your file was transfer correctly
318 1 Gavin Davies
319 1 Gavin Davies
Check the checksum against the tape copy (no json parser installed)
320 1 Gavin Davies
321 1 Gavin Davies
<pre>
322 1 Gavin Davies
# From Database
323 1 Gavin Davies
samweb get-metadata fardet_r00012006_s35_t02.raw | grep "Crc" | cut -d ':' -f 2 | cut -d ' ' -f 2
324 1 Gavin Davies
3828307205
325 1 Gavin Davies
# From file on disk
326 1 Gavin Davies
samweb file-checksum fardet_r00012006_s35_t02.raw | cut -d '"' -f 4
327 1 Gavin Davies
3828307205
328 1 Gavin Davies
</pre>
329 1 Gavin Davies
330 1 Gavin Davies
If you have a json parser available then just use that to parse the output instead of using "cut"
331 1 Gavin Davies
332 1 Gavin Davies
<pre>
333 1 Gavin Davies
samweb get-metadata fardet_r00012006_s35_t02.raw --json | jq '.crc.crc_value'
334 1 Gavin Davies
"3828307205"
335 1 Gavin Davies
</pre>
336 1 Gavin Davies
337 1 Gavin Davies
h1. Advanced Recipes (Hollandaise sauce)
338 1 Gavin Davies
339 1 Gavin Davies
h2. Pre-stage a dataset
340 1 Gavin Davies
341 1 Gavin Davies
For large datasets that are definitely on tape (i.e. haven't been used for a long time) you will want to pre-stage the data to the dCache area before submitting jobs.
342 1 Gavin Davies
343 1 Gavin Davies
h2. Recovering a project
344 1 Gavin Davies
345 1 Gavin Davies
<pre>
346 1 Gavin Davies
samweb project-recovery -e nova --useFileStatus=0 --useProcessStatus=0 gsdavies-RecoFDGENIE_S14-03-25.sh-20140418_1037
347 1 Gavin Davies
</pre>
348 1 Gavin Davies
which yields:
349 1 Gavin Davies
350 1 Gavin Davies
(snapshot_id 15312 minus (project_name gsdavies-RecoFDGENIE_S14-03-25.sh-20140418_1037 and consumed_status consumed))
351 1 Gavin Davies
352 1 Gavin Davies
<pre>
353 1 Gavin Davies
kx509
354 1 Gavin Davies
samweb create-definition <new_definition_name> "(snapshot_id 15312 minus (project_name gsdavies-RecoFDGENIE_S14-03-25.sh-20140418_1037 and consumed_status consumed))"
355 1 Gavin Davies
</pre>
356 1 Gavin Davies
357 1 Gavin Davies
h2. Constructing a Good Runs List
358 1 Gavin Davies
359 1 Gavin Davies
360 1 Gavin Davies
h1. Combining it all (Eggs Benedict)