Project

General

Profile

Data Files from NDOS » History » Version 9

« Previous - Version 9/11 (diff) - Next » - Current version
Brian Rebel, 11/19/2012 03:12 PM


Data Files from NDOS

CHECK WITH THE RUN COORDINATOR BEFORE DOING ANY OF THIS!

This wiki entry describes how to collect data from NDOS and store it as a data file that can be used with our DDT framework.

First, you need to make sure that data is being collected. Check the MicroSlice rate of the DCMs on Ganglia , it should be 20 kHz.

Once that is verified you can log into one of the NDOS buffer nodes:

ssh novadaq@novadaq-ctrl-farm-01.fnal.gov

NDOS has 16 buffer nodes (01 - 16). Note that the novaddt user does not have permissions on the buffer nodes.

Now you just need to set up the environment and run the job:

setup_novaddt
ddt-filter -c ddt-writeFile.fcl

This will run on the data stream and dump the data into a file called ddt.root.

Files from live data should be stored in

/nova/ana/trigger/data/

with format

DDTData-<det>-<size>-<date>-<run>.root
where
  • <det> is one of NDOS, ND, or FD
  • <size> is number of milislices (1000,10000,etc)
  • <date> is of format yymmdd
  • <run> is 1, 2, 3, 4, etc.

Running on Several Buffer Nodes

When you run on several nodes, make sure that you do not write to the shared disk because the parallel jobs will then write to the same location. You should write to an area that is local to the buffer nodes (e.g. /scratch/mfrank).

I logged into ten buffer nodes and executed the following command from each one:

cd /scratch/mfrank/ && setup_novaddt && time ddt-filter -c ddt-writeFile.fcl

There must be an easier way using rgang, but I could not get that quite to work. Here is what I tried from novadaq-ctrl-master:

rgang.py --rsh=ssh 'novadaq-ctrl-farm-{01-8}' 'setup_novaddt;cd /scratch/mfrank;pwd;ddt-filter -c ddt-writeFile.fcl'
I believe rgang does not load the bashrc, so the command setup_novaddt is not known, I might just have to include the pertinent commands from that setup script manually.

Here is a nice clean-up script that works, you can use this after you copied the data to the virtual machines:

rgang.py --rsh=ssh 'novadaq-ctrl-farm-{01-10}' 'cd /scratch/mfrank;rm ddt.root; ls -lh'