Project

General

Profile

Expert Procedures

Powering on/off the detector

Synoptics control of the detector supplies has been flaky (and had cross talk with the ND). I lieu of using synoptics there are scripts that can (should) be used.

To power on:
  1. Log in: novadaq@novadaq-test-master
  2. To power on the LV (24 and 3.5 V) do: power-on-LV_all.sh (for just 1/3, use power_on-LV_dcm-5-01-0(1/3).sh)
  3. To power on the HV do: power-on-HV_all.sh <Volt> (for just 1/3, use power_on-HV_dcm-5-01-0(1/3).sh)
    • You must supply a voltage for the HV script (typically use 425)
To power off:
  1. Log in: novadaq@novadaq-test-master
  2. setup_offline
  3. To power off the HV do: power-off-HV_all.sh
    • The HV needs to drop to less than 200V before turning off the LV
    • The HV voltage fall rate is 5 V/s so you should wait ~45s before powering off the LV (or be extra safe and wait 1.5 minutes for the HV to come all the way down)
  4. To power off the LV (24 and 3.5 V) do: power-off-LV_all.sh
  • Alternatively there are icons on cr-01 that can be use to power off and monitor the state of the power supplies
    • Again the HV gets turned off first then wait until it drops sufficiently
    • Use the "Check Power Supplies" icon to monitor (it is not updating so you will need to run it again when you want to update)

There are also power-<on/off>-<LV/HV>_dcm-5-01-XX.sh scripts that can be used for individual DCMs

Power cycling the MWPC controller

The MWPC controller has been known to get into a bad state, albeit not very often (British sarcasm), e.g. with the timestamps between the controller and the TDCs no longer align. More detailed information on this page: MWPC Configuration and Registers; this page contains just relevant information for power cycling.

To log into the MWPC controller: from novabeamlinedaq00

telnet ftbfwc03 5002

(the DAQ uses port 5001 so don't use that one).

To power cycle the TDCs:

pwrrst

To power cycling the controller: this requires logging into the remote power strip in the rack. This can be done on the Firefox machines on the VNCs, or by creating an SSH tunnel on your local machine.
To do the latter, first set up a tunnel:

ssh -D <port> -f -C -q -N novadaq@novabeamlinedaq00.fnal.gov

where you should pick a local port.
Then configure your browser with a SOCKS proxy. In Firefox, do this in preferences, go into network settings, change proxy from 'No proxy' to 'Manual proxy configuration' and enter localhost as the host, along with the port you opened up the tunnel on.
You can then navigate to ftbf-pdu08.fnal.gov, log in (details in docdb-38595) and power cycling the MWPC controller.

Fixing problems with TDC registers

If power cycling the controller doesn't fix things, individual TDCs may have problems. This may be associated with trigger mis-match errors in the messageViewer.

Log onto the controller from beamlinedaq00:

telnet ftbfwc03 5000

To see status, execute

p0

and to look at the counts on the TDCs, execute
hc 0 rdi 3a

A TDC may be bad if it has particular high counts, >1000 outside of a spill window (they're reset every 10s), or has a bad status (anything other than 2 or 8). In this case, try reloading the firmware for that TDC:
hc <tdc number> ff

(flash firmware). The controller counters can be reset using
wr 1 e4

and then normal behavior can be confirmed again.
It is also possible to use
hc <tdc number> reset

to provide a slightly nicer reset to an individual TDC.

Controlling SiPM ToF detectors

To power on or off:

ssh -Y novadaq@novadaq-test-master.fnal.gov
. power-on-SiPM.sh
. power-off-SiPM.sh

To keep an eye on voltages and currents, open a VNC session:

ssh -L 5943:localhost:5953 -N -f -l novacr03 novadaq-test-master.fnal.gov

If the VNC session on novadaq-test-master.fnal.gov is not already displaying the SiPM voltages, point a browser to:

http://10.226.15.16/

Fixing the TDU timestamp making-and-saving

If the beamline DAQ is no longer processing TDU timestamps, there are many things which could be causing this!

First, ensure SpillServer and the SpillForwarder are running (beamline versions). If they are, the most likely cause is the BoardReader in the beamline DAQ. This can be confirmed by looking at the log files to ensure spills are still being made and forwarded correctly.

Potential reason no TDU timestamps being saved by BoardReader (untested): the address is already being used. To fix, log onto beamline DAQ, run

netstat -ntpl
and look for a process named something like mpirun. Kill it! Mwahaha!

Fixing FTS issues

Mounting beamline /daqdata disk on datadisk

Log into novadaq-test-datadisk.fnal.gov as root and execute

 mount -t nfs 192.168.140.19:/daqdata /beamlinedata/ 

Recovering from filled daqlogs disk

FTS appears to be struggling to keep up with the amount of data we're creating and occasionally (~weekly?) it crashes. This also pulls down DDS with it. This often manifests as Run Control being disconnected from server.

The temporary solution is to manually remove all log files which are old enough to have been copied across.

To list all files older than 5 days do:
cd /daqlogs/TestBeam/Partition1
find ./*/*/* -mtime +5 -exec ls {} \;

To confirm that these have been transferred pick a few of the newest files then in a gpvm do:
samweb list-files "file_name like <filename.log> and availability: virtual"

Finally to remove old files (except dds logs) do:
cd /daqlogs/TestBeam/Partition1
find ./*/*/* -mtime +5 -exec ls {} \; | grep -v dds | xargs rm

This should clear up enough space in /daqlogs and will mean we can start DDS again and connect run control.

Make sure the process is running on each of the relevant machines (http://nusoft.fnal.gov/nova/expert/manual/manual.php?topic=messageviewer). Also, the file

/daqlogs/TestBeam/Partition1/RunControl/novadaq-test-runcontrol/.initRC.xml

gets created as an empty file. You finally need to find an older .initRC_<date>.xml file that isn't 0 size and cp it to .initiRC.xml.

TDU Status Page issues

In general, the first thing to try if the TDU Status Page is showing any issues is just double-clicking the TCR Monitor Restart icon on VNC-01. If that is not sufficient, TDU Web may need restarting as well. To restart TDU Web, log on to novadaq-test-master as novadaq and run:

/home/novadaq/DAQOperationsTools/bin/startTDUWeb-testbeam.sh -z 0

To restart the actual web server (usually only necessary if the DAQ cluster has been restarted) logon to novadaq-test-master as root and do:
service httpd start

Restarting VNC server

On novabeamlinedaq00:

./home/nfs/novadaq/bin/start-vnc-server.sh

On novadaq-test-master: