Project

General

Profile

04 07 14 » History » Version 1

Gavin Davies, 04/24/2014 07:40 PM

1 1 Gavin Davies
h1. 04/07/2014
2 1 Gavin Davies
3 1 Gavin Davies
SAM Meeting Minutes:
4 1 Gavin Davies
Nate, Gavin, Jan, Michael G, Dominick, Chris, Eric, Robert I, Andrew, Peter, Ryan, Craig, Adam
5 1 Gavin Davies
6 1 Gavin Davies
8 to 10 days for small percentage (up to 10%) of priority production streams
7 1 Gavin Davies
8 1 Gavin Davies
FTS Backlog:
9 1 Gavin Davies
Request to purge issued
10 1 Gavin Davies
Robert I update:
11 1 Gavin Davies
33k files deleted (raw2root)
12 1 Gavin Davies
Uncounted from calibration
13 1 Gavin Davies
12k are left
14 1 Gavin Davies
8k are errors
15 1 Gavin Davies
4k are waiting for confirmation that they made it to tape
16 1 Gavin Davies
Queue is empty
17 1 Gavin Davies
18 1 Gavin Davies
Path forward:
19 1 Gavin Davies
FTS improvements implemented
20 1 Gavin Davies
Slower fs easier to handle
21 1 Gavin Davies
MetadataExtractor is hacked, don't use as is
22 1 Gavin Davies
Need to update to latest ART version to pick up new extraction tools
23 1 Gavin Davies
XRootD is still untested, by us, need to get ART on data management areas 
24 1 Gavin Davies
25 1 Gavin Davies
350 to 400 files and hour, will work for 10% sample size we're planning
26 1 Gavin Davies
BUT need to coordinate, SAM Gatekeeper needed for the near future
27 1 Gavin Davies
28 1 Gavin Davies
Ryan:
29 1 Gavin Davies
Biggest fire is calibration constants, Chris needs to be able to declare his files to SAM, so he can run over them. 
30 1 Gavin Davies
31 1 Gavin Davies
FTS completion time ~30 hours
32 1 Gavin Davies
Run over them by hand just like MC to get the constants would probably be faster.
33 1 Gavin Davies
Using the BlueArc drop box system would just land us where we were a week ago, out of disk space, and the throughput would be the same.
34 1 Gavin Davies
35 1 Gavin Davies
Gavin:
36 1 Gavin Davies
Start with 100 files of FD cosmics then scale up by factors of 2 
37 1 Gavin Davies
38 1 Gavin Davies
In the future, should think about running multiple stages in the same job (raw2root, reco, pid), but aren't ready yet with reco stages. This cuts down on intermediary files and sending files over the network.
39 1 Gavin Davies
40 1 Gavin Davies
Chris will run data calibration constants manually, not through SAM.
41 1 Gavin Davies
42 1 Gavin Davies
Michael is the data managing contact. He is to be contacted of any ramp up. He will be in WH12XO in Andrew's office.
43 1 Gavin Davies
44 1 Gavin Davies
dCache dropbox is ready for slow submission. FTS restart needed when dCache drop box is configured. Gavin will configure the drop box.
45 1 Gavin Davies
46 1 Gavin Davies
Monitoring:
47 1 Gavin Davies
FTS status page, best resource
48 1 Gavin Davies
Can look at enstore dcache pages, not that helpful
49 1 Gavin Davies
Michael is a good contact here as well
50 1 Gavin Davies
51 1 Gavin Davies
Need more people to help monitoring.
52 1 Gavin Davies
53 1 Gavin Davies
Action Items:
54 1 Gavin Davies
Chris runs data calibration constants manually
55 1 Gavin Davies
Gavin configures dCache dropbox
56 1 Gavin Davies
Dominick to help with MC reco processing
57 1 Gavin Davies
Luke will run absolute calibration for data after Chris is done
58 1 Gavin Davies
New tag
59 1 Gavin Davies
Batch data production to begin
60 1 Gavin Davies
61 1 Gavin Davies
Database issues still a mystery
62 1 Gavin Davies
Low on space on Bluearc, needs addressing soon
63 1 Gavin Davies
Jan and Pavan to start coordinating today
64 1 Gavin Davies
65 1 Gavin Davies
Gavin is SAM Gatekeeper
66 1 Gavin Davies
67 1 Gavin Davies
Educating users is important. Discussion about tutorial and best practices. 
68 1 Gavin Davies
Need to follow up offline.