Current Running Conditions Temporary Conditions New & Permanent Conditions Pay Special Attention

Please make sure that you are familiar with the SHIFT Bulletin Board.

Current release:

We are running DAQ R16 on both FD and ND!!!

February 2020: Updated Kill Partition script

There is an updated version of the Kill Partition script available. To run it, logon to master and execute

/home/novadaq/testRel_FDOps_20200211/DAQOperationsTools/script/ -z 1 -m -d FarDet

You can also pass this script the -r flag to have it show you what is still running.

Starting ND supernova noise map process (updated Dec. 5, 2018)

On Nov. 24, 2018, there was a rise in the dew point underground leading to higher ND noise level. As a result, there were bursts of supernova triggers induced by the higher noise level. Later Andrey found it was actually due to the noise map module being terminated for unknown reasons. If you ever want to restart the noise map process, which is a non-disruptive operation, please follow this elog entry:

Nov. 20th - Verify if killPartition1 script killed all DAQ processes in Partition 1

NOTE: This script does not currently exist (30 July 2019) - we lost it in the FD RAID array failure and it still needs rewriting.

After executing killPartition script, please run the following script as novadaq on any FarDet node (preferably novadaq-far-master) to verify if there are still any running DAQ processes related to Partition 1:


Permanent: Instructions for resizing a control room VNC

If someone connects to one of the VNCs remotely and the window resizes, the instructions for resizing the VNC window are here

Permanent: No terminals open on ND CR-05 (ND run control) ("new" Oct. 2017)

Please make sure there are no terminals windows open on this machine unless you or another expert is currently doing work or you have been instructed by someone to leave it open. This includes checking to make sure windows are not minimized. Simply keep typing "exit" until the windows disappear. Having a terminal window open prevents the KillPartition script from working.

Update on 01/17/2017 High Temperature Event

  • Procedures to follow during a temperature incident are here*

Please check if the loadshed script is running by checking the timestamp of the last log entry in the /var/log/loadshed.log file on The timestamp should be within 5 minutes of the current time.

ssh -l novadaq "tail /var/log/loadshed.log" 

Examples outputs look like:

Wed Jan 18 15:05:01 CST 2017: R4I: W, R4E: X, R9I: Y, R9E: Z
Tempsensors: W: 0, E: 0, C: 0, Z: 0, T: 4
Farm: W: 1, E: 1, C: 1, Z: 0, T: 128
Wed Jan 18 15:05:06 CST 2017: Current Errorlevel is: 0

Where the numbers in the first line show the difference between the current temperature and the baseline temperature in the unit of 0.01F. The baseline temperatures are


Hence, the first line can be interpreted as:

Wed Jan 18 15:05:01 CST 2017: Amount temps away nominal (degrees f): R4I: -2.64, R4E: -3.13, R9I: -3.34, R9E: -3.93

The second line and the third line represents the following:


  • If you need to power off DAQ nodes immediately or the loadshed script is not running, you can power off FD DAQ nodes by following this page .*

DSO Reprocessing Node List

If any of these machines die, follow the instructions at the bottom of the DSO Scan instructions.

  • FarDet:
    far-farm-10, 13, 14, 17, 18, 19, 21, 22, 23, 24, 25, 26
  • NearDet:
    near-farm-12, 13, 14

Resource List

  • FarDet:
    Managers(2020-05-03): ConfigurationManager, DAQApplicationManager, DataLogger, EventDispatcher4, GlobalTrigger, MessageAnalyzer, MessageFacilityServer, MessageViewer, RunControlServer, SNEWSMessage, SpillServer, TDUManager, TriggerScalars4
    BNEVB List (2020-09-08): 10-47 excluding 11, 12, 15, 16, 20, 28, 35, and 56-200 excluding 57, 59, 61, 78, 92, 96, 114, 115, 116, 126, 130, 131, 133, 142, 150, 164, 170, 180, 181, 184, 189.
    Buffer Nodes ready to return to service: None at this time
    Timing chains(2017-08-04): ALL - DiB-{01-14}{s,t}

Last update on 04/29/2019 Current timing chain in use

FD Chain 2 and ND Chain 2. FD Chain 1 and ND Chain 1 are backup timing chains. tdu-near-master-01 is where SpillServer runs.