Project

General

Profile

Environment Emergency Procedures

We now have an automated mechanism to shutdown detector systems in response to detector environment incidents/alarms. However, it is still up to shifters/experts to ensure APDs and FEBs are not damaged by environmental parameters going out of the nominal detector operation window. Below are monitoring and response actions to be taken by the shifter in case of environmental alarms. In most of the cases, the automated actions will protect the detector from damage, but when the FEBs and APDs are energized with HV and LV, at least one shifter needs to be monitoring the detector environment incidents/alarms at all times!

Monitoring detector environment incidents/alarms

  • Every few minutes, glance at the window and check the following:
    1. Timestamp is current (synoptics windows are known to freeze frequently)
    2. Environment dew point is < 10.0 °C (otherwise, FEBs can be damaged by humidity)
    3. Dryer System dew point (AMT25) is < -30 °C (otherwise APDs can be damaged by condensation)
    4. The "Inlet Temp" from the APD Cooling system, under "System Health" is < 20 °C (otherwise APDs will not be able to cool down)
    5. All the boxes under "System Health" should be green. If any of them becomes red, execute immediately the emergency procedure below. Note that violation of the other conditions above may not cause any of the boxes to go red.

Emergency response to detector environment incidents/alarms

When the above conditions are not satisfied, the shifter/expert should initiate immediately (within a maximum of 5 minutes of the occurrence) the following detector shutdown procedure (if the automated system failed to do so):
  • turn off HV on the APDs by using the 'Turn Off APD HV' icon on VNC 1 desktop, shown in picture below.
  • turn off LV for the FEBs by using the 'Turn Off LV' icon on VNC 1 desktop, also shown in picture below
  • Check that HV and LV are off by double-clicking the 'Check Power Supplies' icon on VNC 1 desktop, the output should look like the attached image. In particular check that "HV Supplied Voltage" and "LV Supplied Voltage" values in the right are all ~0 . (note that the synoptic monitoring of HV and LV has worked unreliably, so do not use that for now)
  • Check that APDs are no longer cooled using the CSS GUI
  • Stop the detector run using the rcGUI
  • Contact the relevant expert to recover the system which failed, or follow recovery procedures listed below
  • Document all your actions in the NOvA Test Beam ECL: http://dbweb5.fnal.gov:8080/ECL/novatestbeam/E/index (use services password to log in)

After detector operation environment is back to normal (according to Monitoring section above), powering up the detector should be done by an expert following procedures found in Expert Procedures.


Turning off Water Chiller

  • Even if the detector is fully turned off, If environmental dew point is rising towards and above large values like 13 °C, or if the water temperature is above the 20 °C level, turn off water chiller with the orange toggle switch on the left side of the control panel, shown in picture below
  • Inform the Run Coordinator that the chiller is off, and document all your actions in the NOvA Test Beam ECL.

If the HVAC is not operational, the environmental dew point inside MC7 can quickly rise to values of 13 °C or more. If that value rises above the APD System's inlet temperature (or water chiller outlet temperature), which will typically hover between 14 and 16 °C, condensation will occur along the water systems lines, including on the lines connected to the APDs. This could cause water to drip into the electronics, so it has to be avoided! Also, we have seen in four occasions the water chiller stop cooling water because of a stuck bypass valve (as shown in the picture, where the "chilled" water temperature is 95 °F instead of the correct 62°F).

HVAC recovery

The recently installed HVAC unit occasionally experiences problems where it shuts down and regardless of control thermostat inputs, it will not run again until power-cycled. The following procedure restores it to normal operation, until we get a better understanding of the causes for the shutdowns from the installation company:
  • Under controlled or open access, check that the settings of the control thermostat on the MC7 West wall are all correct (see picture, the green light indicates the thermostat thinks the unit should be running)

  • Go to the outside of MC7, West side, by the HVAC unit. If you are in controlled access (CA), note that you cannot use the MC7b West-side door, or you will drop the interlocks. Follow the same CA procedure you would use to leave the enclosure, but walk along the West side of the building till you see the unit as in the picture

  • Turn off the knife switch energizing the HVAC unit, wait 10 seconds, and turn it back on. (You may need NFPA70E training and protective equipment in order to do this, we are in the process of confirming it)
  • Go back inside MC7, following CA procedures if necessary, and press the "Mode" button until you cycle back to "Cool" mode. The green light should have disappeared. It will come back on in 5-10 minutes, and the HVAC unit will resume operation at that point. You can verify that by noticing the air suction through the duct next to the thermostat. If the unit still did not turn on, turn off the water chiller (see below), and inform the Run Coordinator.
  • Finally, verify that the unit heater on the West side of the enclosure (near the heat tape power cart) is set to operate at a dial temperature between 70-75 °F. This helps prevent the HVAC unit from stopping cooling and shutting down. Do not turn on the East side heater (knife-switch turned off), as it would blow hot air on the detector electronics.
  • Document all your actions in the NOvA Test Beam ECL.

Dry Gas System Troubleshooting