Power Outage Planning

Prep work:

  • System experts need passwords:
  1. PMT: Slow Controls HV power-supply control.
  2. TPC: glassman connection.
  3. CRT: Util crate slowmon
  • We need a DAQ configuration that configures the ASICs. Config 816 is current as of July 30,2019. Take data for any amount of time. Once run starts ASICs are configured. Make sure run does not re-start or you will re-configure the ASICs. This config must be used in expert mode. Once the run crashes, start a new run with the normal config.
  • We need a DAQ config to calibrate the PMTs. Current config (as of 7/30/19) is 808. Make sure this is available before the outage starts.
  • If you want to take data during ramp-up or ramp-down and the PMTs are off you will need a config which has an unbiased trigger and goes to the TestRun folder.
    • If it is not in the TestRun folder, it will cause PUBS errors.

A power-outage schedule to guide planning can be found below. Outages generally start early in the morning, so this guide assumes the power-down procedure will take place the day before. In the plan below


  • The cryo system should be down for as little time as possible to prevent buildup of impurities and a degradation of our electron lifetime.
  • The ASIC LV should be powered up as soon as this is possible. The ASIC LV powers the diodes which protect our electronics, hence the urgency. Likewise, ASIC LV should be one of the last items to be turned off. This should happen after other TPC HV systems are off, and after the PMT HV is off.
  • Before leaving for the day the night before the outage, change the Run Plan instructing shifters to complete the "Power Outage Checklist" rather then the normal one, a streamlined version which forces the shifter to focus on subsystems / items which can still be monitored.
  • Subsystem dependencies:
  1. TPC HV, ASIC LV, and wire-bias HV should come up only after the Cryo system is on.
  2. ASIC LV should be powered down after PMT HV, wire-bias, and TPC HV are powered down.
  3. Data taking in BeamOff or Physics mode requires PMTs to be on. This is because otherwise PUBS will not be able to process incoming files.
  4. If turning on the laser rack, Slow Controls needs to be on before turning the laser key, otherwise the PMT interlock alarm will prevent the laser from turning on.
  5. PMT HV, Wire-bias, TPC HV all need the cryo system back up before being able to ramp back up.

Step-by-step Guide:

Day before outage:

  1. 14:15 Take a 10 minute PMT only run for direct reference when the gain adjustment is done after the outage - PMT expert. (10 minutes)
  2. 14:25 Stop data-taking. Make an elog with last run number with normal run conditions. - shifter (5 min.)
  3. 14:30 Turn off PUBS - DM expert (30 min.)
  4. 14:30 Turn off CRT remotely - CRT expert (30 min.)
  5. 14:30 power down PMT HV - PMT expert (30 min.)
  6. 15:00 ramp down TPC HV - TPC/HV expert (2-3 hours)
  7. 17:00 ramp down wire-bias - TPC/HV expert (1 hour)
  8. 18:00 power off ASIC LV - TPC/HV expert (30 min.)
  9. 18:30 power down racks - RunCo + RO expert. CRT/MuCS expert also may be necessary. (1 hour.)
    a Make sure data-taking has stopped before powering off any TPC/PMT rack.
    b Make sure PMTs are powered off before touching the PMT-Trig crate. Check explicitly with PMT expert via phone.
    c Make sure the TPC wire-bias and ASIC LV are powered off before powering down the TPC-PS rack. Check explicitly with the TPC-HV expert via phone.
    d Make sure the TPC HV is powered down before turning off the TPC HV racks. Check explicitly with the TPC-HV expert via phone.
  10. After all of the above are off the shifter can acknowledge the following alarms:
    • OnDetPower
    • RackFans
    • RackProt (This is for the RPS "Rack Protection Systems")
    • RackTemps
    • ZMON
    • TPCDrift
    • Crate Rails
    • CRT DAQ
    • uB_BeamData_TRIG/Age

This will make it easier to spot if there are any real alarms overnight. Note that for most of these the alarms can be acknowledged by right clicking in the top level of the alarm tree which will acknowledge all of these alarms in a given category at once.

Day of outage, before:

  1. T-30 min. Turn off slow-controls. - SMC expert (10 min.)
  2. T-20 min. Turn off non-evb machines - SLAM team (5 min.)
  3. T-10 min. Turn off evb - SLAM team (5 min.)
  4. T-5 min. Power down cryo - Cryo engineers, remotely (5 min.)

Day of / after outage, power-up:

  1. T+5 min. Power up cryo - Cryo engineers (up to 1 hour)
    a. Make sure Keythleys in TPC-HV rack are configured. See e-log 77738:
  2. T+5 min. Power-up racks in LArTF - RunCo + RO expert (1 hour)
  3. T+5 min. Power-on machines @ LArTF - SLAM team (5 min.)
  4. T+15 min. Power-on slow controls - Slow Controls (30 min.)
  5. T+1 hour. Turn on ASIC LV. - TPC/HV expert (30 min.)
  6. T+1 hour. Turn on PUBS - DM expert (30 min.)
  7. T+1 hour Turn on CRT - CRT expert (30 min.)
  8. T+1 hour Turn on PMTs - PMT expert (30 minutes)
  9. T+1:30 hour Turn on wire-bias HV - TPC/HV expert (1 hour)
  10. T+2:30 hours Ramp up TPC HV - TPC/HV expert (2/3 hours)
  11. Start taking data. First run should use config ID that re-configures the ASICs. This step can be performed once the ASIC LV is up.
  12. PMT Calibration run (30 minutes)
  13. Start taking a normal data run. Make elog with run number of first run with normal run conditions. - shifter

Once ASIC LV is up and ASICs are reconfigured, check the status of the ASICs by taking a run and checking the plot below with the OM:
The first run number (18559 above) is the new run you just took. The second (17524) is the reference, preferably a run taken right before the power-down. If you see substantial differences in a crate associated to a full card (32 or 64) channels try first power-cycling the crate (with RO expert approval) before attributing the difference to a mis-configuration.

Template power outage schedule used summer 2019 is at :