Project

General

Profile

ExpertBulletinBoard

Current Running Conditions Temporary Conditions New & Permanent Conditions Pay Special Attention

Please make sure that you are familiar with the SHIFT Bulletin Board.

Current release:

We are running DAQ R16 on both FD and ND!!!

On March 19, 2018, we rolled out an entirely new set of shift forms. Descriptions of these forms as well as a schedule for completing them can be found at -
Docdb-27620.

Starting ND supernova noise map process (updated Dec. 5, 2018)

On Nov. 24, 2018, there was a rise in the dew point underground leading to higher ND noise level. As a result, there were bursts of supernova triggers induced by the higher noise level. Later Andrey found it was actually due to the noise map module being terminated for unknown reasons. If you ever want to restart the noise map process, which is a non-disruptive operation, please follow this elog entry:
http://dbweb6.fnal.gov:8080/ECL/nova/E/show?e=142811

Dec. 1 - Switched FD datadisk back to datadisk-05

Having cleared up the issues with the instabilities for datadisk-05, we have switched back to longterm running on datadisk-05. This is likely permanent. For the ECL entry with instructions about what was changed, see here

Nov. 20th - Verify if killPartition1 script killed all DAQ processes in Partition 1

NOTE: This script does not currently exist (30 July 2019) - we lost it in the FD RAID array failure and it still needs rewriting.

After executing killPartition script, please run the following script as novadaq on any FarDet node (preferably novadaq-far-master) to verify if there are still any running DAQ processes related to Partition 1:

/home/novadaq/P1_Processes/get_all_proc.sh

Permanent: Instructions for resizing a control room VNC

If someone connects to one of the VNCs remotely and the window resizes, the instructions for resizing the VNC window are here

Permanent: No terminals open on ND CR-05 (ND run control) ("new" Oct. 2017)

Please make sure there are no terminals windows open on this machine unless you or another expert is currently doing work or you have been instructed by someone to leave it open. This includes checking to make sure windows are not minimized. Simply keep typing "exit" until the windows disappear. Having a terminal window open prevents the KillPartition script from working.

Update on 01/17/2017 High Temperature Event

  • Procedures to follow during a temperature incident are here*

Please check if the loadshed script is running by checking the timestamp of the last log entry in the /var/log/loadshed.log file on novadaq-far-master.fnal.gov. The timestamp should be within 5 minutes of the current time.

ssh -l novadaq novadaq-far-master.fnal.gov "tail /var/log/loadshed.log" 

Examples outputs look like:

Wed Jan 18 15:05:01 CST 2017: R4I: W, R4E: X, R9I: Y, R9E: Z
Tempsensors: W: 0, E: 0, C: 0, Z: 0, T: 4
Farm: W: 1, E: 1, C: 1, Z: 0, T: 128
Wed Jan 18 15:05:06 CST 2017: Current Errorlevel is: 0

Where the numbers in the first line show the difference between the current temperature and the baseline temperature in the unit of 0.01F. The baseline temperatures are

RACK_04_INTERNAL_BASELINE=6300
RACK_04_EXTERNAL_BASELINE=8000
RACK_09_INTERNAL_BASELINE=6300
RACK_09_EXTERNAL_BASELINE=7000

Hence, the first line can be interpreted as:

 
Wed Jan 18 15:05:01 CST 2017: Amount temps away nominal (degrees f): R4I: -2.64, R4E: -3.13, R9I: -3.34, R9E: -3.93

The second line and the third line represents the following:

    echo "Tempsensors: W: $NUM_WARNING, E: $NUM_ERROR, C: $NUM_CRITICAL, Z: $NUM_CRAZY, T: $TEMPSENSOR_COUNT" 
    echo "Farm: W: $FARM_NUM_WARNING, E: $FARM_NUM_ERROR, C: $FARM_NUM_CRITICAL, Z: $FARM_NUM_CRAZY, T: $NODE_COUNT" 

  • If you need to power off DAQ nodes immediately or the loadshed script is not running, you can power off FD DAQ nodes by following this page .*

Resource List

  • FarDet:
    Managers(2018-08-08): ConfigurationManager, DAQApplicationManager, DataLogger, EventDispatcher5, GlobalTrigger, MessageAnalyzer, MessageFacilityServer, MessageViewer, RunControlServer, SNEWSMessage, SpillServer, TDUManager, TriggerScalars5
    BNEVB List (2019-10-09): 10-47 excluding 12, 16, 28, 29, and 56-200 excluding 57, 59, 61, 62, 77, 78, 92, 126, 133, 164, 180, 181, 184.
    Timing chains(2017-08-04): ALL - DiB-{01-14}{s,t}

Last update on 04/29/2019 Current timing chain in use

FD Chain 2 and ND Chain 2. FD Chain 1 and ND Chain 1 are backup timing chains. tdu-near-master-01 is where SpillServer runs.