Project

General

Profile

Data Files from NDOS » History » Version 7

Martin Frank, 11/05/2012 03:02 PM

1 1 Martin Frank
h1. Data Files from NDOS
2 1 Martin Frank
3 4 Martin Frank
*CHECK WITH THE RUN COORDINATOR BEFORE DOING ANY OF THIS!*
4 4 Martin Frank
5 1 Martin Frank
This wiki entry describes how to collect data from NDOS and store it as a data file that can be used with our DDT framework.
6 2 Martin Frank
7 3 Martin Frank
First, you need to make sure that data is being collected.  Check the MicroSlice rate of the DCMs on "Ganglia":https://novadaq-ctrl-datamon.fnal.gov/ganglia/?r=2hr&cs=&ce=&m=MicroSlice%2520Rate_P0&s=by+name&P=0&c=DCM&h=&host_regex=&max_graphs=0&sh=1&z=small&hc=4&q= , it should be 20 kHz.
8 1 Martin Frank
9 3 Martin Frank
Once that is verified you can log into one of the NDOS buffer nodes:
10 3 Martin Frank
11 2 Martin Frank
<pre>
12 2 Martin Frank
ssh novadaq@novadaq-ctrl-farm-01.fnal.gov
13 2 Martin Frank
</pre>
14 2 Martin Frank
15 2 Martin Frank
NDOS has 16 buffer nodes (01 - 16).  Note that the @novaddt@ user does not have permissions on the buffer nodes.
16 2 Martin Frank
17 2 Martin Frank
Now you just need to set up the environment and run the job:
18 2 Martin Frank
19 1 Martin Frank
<pre>
20 2 Martin Frank
setup_novaddt
21 3 Martin Frank
ddt-filter -c ddt-writeFile.fcl
22 2 Martin Frank
</pre>
23 2 Martin Frank
24 3 Martin Frank
This will run on the data stream and dump the data into a file called @ddt.root@.
25 5 Jan Zirnstein
26 5 Jan Zirnstein
Files from live data should be stored in
27 5 Jan Zirnstein
28 5 Jan Zirnstein
<pre>
29 5 Jan Zirnstein
/nova/ana/trigger/data/
30 5 Jan Zirnstein
</pre>
31 5 Jan Zirnstein
32 5 Jan Zirnstein
with format
33 5 Jan Zirnstein
34 5 Jan Zirnstein
<pre>
35 5 Jan Zirnstein
DDTData-<det>-<size>-<date>-<run>.root
36 5 Jan Zirnstein
</pre>
37 5 Jan Zirnstein
38 5 Jan Zirnstein
where 
39 5 Jan Zirnstein
* <det> is one of NDOS, ND, or FD
40 5 Jan Zirnstein
* <size> is number of milislices (1000,10000,etc)
41 5 Jan Zirnstein
* <date> is of format yymmdd
42 5 Jan Zirnstein
* <run> is 1, 2, 3, 4, etc.
43 6 Martin Frank
44 6 Martin Frank
h2. Running on Several Buffer Nodes
45 6 Martin Frank
46 6 Martin Frank
When you run on several nodes, make sure that you do not write to the shared disk because the parallel jobs will then write to the same location.  You should write to an area that is local to the buffer nodes (e.g. @/scratch/mfrank@).  
47 6 Martin Frank
48 6 Martin Frank
I logged into ten buffer nodes and executed the following command from each one:
49 6 Martin Frank
<pre>
50 6 Martin Frank
cd /scratch/mfrank/ && setup_novaddt && time ddt-filter -c ddt-writeFile.fcl
51 6 Martin Frank
</pre>
52 6 Martin Frank
53 6 Martin Frank
There must be an easier way using @rgang@, but I could not get that quite to work.  Here is what I tried from @novadaq-ctrl-master@:
54 6 Martin Frank
<pre>
55 6 Martin Frank
rgang.py --rsh=ssh 'novadaq-ctrl-farm-{01-8}' 'setup_novaddt;cd /scratch/mfrank;pwd;ddt-filter -c ddt-writeFile.fcl'
56 6 Martin Frank
</pre>I believe @rgang@ does not load the bashrc, so the command setup_novaddt is not known, I might just have to include the pertinent commands from that setup script manually. 
57 7 Martin Frank
58 7 Martin Frank
Here is a nice clean-up script that works:
59 7 Martin Frank
<pre>
60 7 Martin Frank
rgang.py --rsh=ssh 'novadaq-ctrl-farm-{01-10}' 'cd /scratch/mfrank;rm ddt.root; ls -lh'
61 7 Martin Frank
</pre>