Initial 35t DAQ cluster setup

Week of 04-Aug-2014, Erik Blaufaus, Tim Nicholls, Matthew Thorpe, John Freeman, Kurt Biery

Node names

  • - gateway node, has access to both campus and private network
  • lbnedaq1 - DAQ node, private network access only
  • lbnedaq2 - DAQ node, private network access only
  • lbnedaq3 - DAQ node, private network access only

At the moment, we are primarily using a shared account named lbnedaq on these four nodes.

Locations of relevant files

Central areas:
  • Log files: subdirectories under /data/lbnedaq/daqlogs
  • Data: /data/lbnedaq/data
  • UPS Products: /data/lbnedaq/products
    • This area is not being used heavily at the moment. Each of the code distribution areas has its own products area with all of the UPS product that it needs. But, we expect to migrate to using the central products area over time.
Code distributions/work areas:
  • /data/lbnedaq/rce
    • This area has the version of the lbne_artdaq code that is being developed for the RCE readout. It has the RCE emulator that has been developed by Tim and Matt.
  • /data/lbnedaq/ssp
    • This area has the version of the lbne_artdaq code that is being developed for the SSP readout. It includes the latest changes to the lbne_artdaq package to split off the raw data classes.

DAQ node configuration notes (Tim)


On each DAQ node:

groupadd --gid 9960 lbne
useradd -u 49491 -g 9960 -k /etc/skel -m -s /bin/bash lbnedaq
passwd lbnedaq -> <<daq password>>
Copy SSH pub/priv key pair over from gateway and set up

Fix NFS ID mapping squash by setting domain = in /etc/idmapd.conf and running nfsidmap -c
Configured 10GigE interfaces (p4p1) on each machine 10.0.1.x/24, nodes 2 and 3 are connected by fibre
Needed to install ruby on DAQ nodes to run artdaq - manual install of ruby, ruby-libs, compat-readline5 RPMs as no yum access
Disabled iptables on DAQ nodes


Renamed machines to lbnedaq1, lbnedaq3, lbnedaq3, updated hosts files etc
Updated timezone to America/Chicago in /etc/sysconfig/clock, copied /usr/share/timezone/America/Chicago to /etc/localtime
Configured ntpd to sync with gateway and started service
Installed tmux on lbnedaq1-3 to allow detached process management
Blacklisted ftdi_sio driver on lbnedaq1 and installed /etc/udev/rules.d/99-lbne-devices.rules to allow USB access to SSP module
Distributed MPI across system requires rsh for mpirun launcher. Installed rsh, rsh-server, xinetd on lbnedaq[1-3]. Enabled services in /etc/xinetd.d/[rsh,rlogin,rexec] and created /etc/hosts.equiv file as appropriate


Added extra rule to /etc/udev/rules.d/99-lbne-devices.rules to set group on USB interface to NOvA TDU - appears on /dev/ttyUSB0
Installed dhcpd and configured on lbnedaq1 to allow TDU to pick up IP address configuration
Installed DHCPD on lbnedaq1 to allow NOvA TDU to get IP address -


Installed zeromq, zeromq-devel and openpgm at request of Erik to support Run Control, DAQinterface etc

Gateway RPMs installed