Project

General

Profile

Data Files from NDOS » History » Version 9

Brian Rebel, 11/19/2012 03:12 PM

1 9 Brian Rebel
{{toc}}
2 9 Brian Rebel
3 1 Martin Frank
h1. Data Files from NDOS
4 1 Martin Frank
5 4 Martin Frank
*CHECK WITH THE RUN COORDINATOR BEFORE DOING ANY OF THIS!*
6 4 Martin Frank
7 1 Martin Frank
This wiki entry describes how to collect data from NDOS and store it as a data file that can be used with our DDT framework.
8 2 Martin Frank
9 3 Martin Frank
First, you need to make sure that data is being collected.  Check the MicroSlice rate of the DCMs on "Ganglia":https://novadaq-ctrl-datamon.fnal.gov/ganglia/?r=2hr&cs=&ce=&m=MicroSlice%2520Rate_P0&s=by+name&P=0&c=DCM&h=&host_regex=&max_graphs=0&sh=1&z=small&hc=4&q= , it should be 20 kHz.
10 1 Martin Frank
11 3 Martin Frank
Once that is verified you can log into one of the NDOS buffer nodes:
12 3 Martin Frank
13 2 Martin Frank
<pre>
14 2 Martin Frank
ssh novadaq@novadaq-ctrl-farm-01.fnal.gov
15 2 Martin Frank
</pre>
16 2 Martin Frank
17 2 Martin Frank
NDOS has 16 buffer nodes (01 - 16).  Note that the @novaddt@ user does not have permissions on the buffer nodes.
18 2 Martin Frank
19 2 Martin Frank
Now you just need to set up the environment and run the job:
20 2 Martin Frank
21 1 Martin Frank
<pre>
22 2 Martin Frank
setup_novaddt
23 3 Martin Frank
ddt-filter -c ddt-writeFile.fcl
24 2 Martin Frank
</pre>
25 2 Martin Frank
26 3 Martin Frank
This will run on the data stream and dump the data into a file called @ddt.root@.
27 5 Jan Zirnstein
28 5 Jan Zirnstein
Files from live data should be stored in
29 5 Jan Zirnstein
30 5 Jan Zirnstein
<pre>
31 5 Jan Zirnstein
/nova/ana/trigger/data/
32 5 Jan Zirnstein
</pre>
33 5 Jan Zirnstein
34 5 Jan Zirnstein
with format
35 5 Jan Zirnstein
36 5 Jan Zirnstein
<pre>
37 5 Jan Zirnstein
DDTData-<det>-<size>-<date>-<run>.root
38 5 Jan Zirnstein
</pre>
39 5 Jan Zirnstein
40 5 Jan Zirnstein
where 
41 5 Jan Zirnstein
* <det> is one of NDOS, ND, or FD
42 5 Jan Zirnstein
* <size> is number of milislices (1000,10000,etc)
43 5 Jan Zirnstein
* <date> is of format yymmdd
44 5 Jan Zirnstein
* <run> is 1, 2, 3, 4, etc.
45 6 Martin Frank
46 6 Martin Frank
h2. Running on Several Buffer Nodes
47 6 Martin Frank
48 6 Martin Frank
When you run on several nodes, make sure that you do not write to the shared disk because the parallel jobs will then write to the same location.  You should write to an area that is local to the buffer nodes (e.g. @/scratch/mfrank@).  
49 6 Martin Frank
50 6 Martin Frank
I logged into ten buffer nodes and executed the following command from each one:
51 6 Martin Frank
<pre>
52 6 Martin Frank
cd /scratch/mfrank/ && setup_novaddt && time ddt-filter -c ddt-writeFile.fcl
53 6 Martin Frank
</pre>
54 6 Martin Frank
55 6 Martin Frank
There must be an easier way using @rgang@, but I could not get that quite to work.  Here is what I tried from @novadaq-ctrl-master@:
56 6 Martin Frank
<pre>
57 6 Martin Frank
rgang.py --rsh=ssh 'novadaq-ctrl-farm-{01-8}' 'setup_novaddt;cd /scratch/mfrank;pwd;ddt-filter -c ddt-writeFile.fcl'
58 6 Martin Frank
</pre>I believe @rgang@ does not load the bashrc, so the command setup_novaddt is not known, I might just have to include the pertinent commands from that setup script manually. 
59 7 Martin Frank
60 8 Martin Frank
Here is a nice clean-up script that works, you can use this after you copied the data to the virtual machines:
61 7 Martin Frank
<pre>
62 7 Martin Frank
rgang.py --rsh=ssh 'novadaq-ctrl-farm-{01-10}' 'cd /scratch/mfrank;rm ddt.root; ls -lh'
63 7 Martin Frank
</pre>