Power Outage Planning » History » Version 17
David Caratelli, 08/27/2018 01:37 PM
h1. Power Outage Planning
h2. Prep work:
* System experts need passwords:
> # PMT: Slow Controls HV power-supply control.
> # TPC: glassman connection.
* We need a DAQ configuration that configures the ASICs. Examples from the past is config 768. Take data for any amount of time. Once run starts ASICs are configured. Make sure run does not re-start or you will re-configure the ASICs.
* We need a DAQ config to calibrate the PMTs. Current config is 792. Make sure this is available before the outage starts.
* If you want to take data during ramp-up or ramp-down and the PMTs are off you will need a config which has an unbiased trigger and goes to the TestRun folder.
** If it is not in the TestRun folder, it will cause PUBS errors.
A power-outage schedule to guide planning can be found below. Outages generally start early in the morning, so this guide assumes the power-down procedure will take place the day before. In the plan below
* The cryo system should be down for as little time as possible to prevent buildup of impurities and a degradation of our electron lifetime.
* The ASIC LV should be powered up as soon as this is possible. The ASIC LV powers the diodes which protect our electronics, hence the urgency. Likewise, ASIC LV should be one of the last items to be turned off. This should happen after other TPC HV systems are off, and after the PMT HV is off.
* Rack power-up/down instructions are on the [[Rack Power ONOFF procedures]] page.
* Subsystem dependencies:
> # TPC HV, ASIC LV, and wire-bias HV should come up only after the Cryo system is on.
> # ASIC LV should be powered down after PMT HV, wire-bias, and TPC HV are powered down.
> # Data taking in BeamOff or Physics mode requires PMTs to be on. This is because otherwise PUBS will not be able to process incoming files.
> # If turning on the laser rack, Slow Controls needs to be on before turning the laser key, otherwise the PMT interlock alarm will prevent the laser from turning on.
> # PMT HV, Wire-bias, TPC HV all need the cryo system back up before being able to ramp back up.
h2. Step-by-step Guide:
Day before outage:
# 14:25 Stop data-taking - shifter (5 min.)
# 14:30 Turn off PUBS - DM expert (30 min.)
# 14:30 Turn off CRT remotely - CRT expert (30 min.)
# 14:30 power down PMT HV - TPC/HV expert (30 min.)
# 15:00 ramp down TPC HV - TPC/HV expert (2-3 hours)
# 17:00 ramp down wire-bias - TPC/HV expert (1 hour)
# 18:00 power off ASIC LV - TPC/HV expert (30 min.)
# 18:30 power down racks - RunCo + RO expert. CRT/MuCS expert also may be necessary. (1 hour.)
a Make sure data-taking has stopped before powering off any TPC/PMT rack.
b Make sure PMTs are powered off before touching the PMT-Trig crate. Check explicitly with PMT expert via phone.
c Make sure the TPC wire-bias and ASIC LV are powered off before powering down the TPC-PS rack. Check explicitly with the TPC-HV expert via phone.
d Make sure the TPC HV is powered down before turning off the TPC HV racks. Check explicitly with the TPC-HV expert via phone.
Day of outage, before:
# T-30 min. Turn off slow-controls. - SMC expert (10 min.)
# T-20 min. Turn off non-evb machines - SLAM team (5 min.)
# T-10 min. Turn off evb - SLAM team (5 min.)
# T-5 min. Power down cryo - Cryo engineers, remotely (5 min.)
Day of / after outage, power-up:
# T+5 min. Power up cryo - Cryo engineers (up to 1 hour)
a. Make sure Keythleys in TPC-HV rack are configured. See e-log 77738: http://dbweb5.fnal.gov:8080/ECL/uboone/E/show?e=77738.
# T+5 min. Power-up racks in LArTF - RunCo + RO expert (1 hour)
# T+5 min. Power-on machines @ LArTF - SLAM team (5 min.)
# T+15 min. Power-on slow controls - Slow Controls (30 min.)
# T+1 hour. Turn on ASIC LV. - TPC/HV expert (30 min.)
# T+1 hour. Turn on PUBS - DM expert (30 min.)
# T+1 hour Turn on CRT - CRT expert (30 min.)
# T+1:30 hour Turn on wire-bias HV - TPC/HV expert (1 hour)
# T+2:30 hours Ramp up TPC HV - TPC/HV expert (2/3 hours)
# Start taking data. First run should use config ID that re-configures the ASICs. This step can be performed once the ASIC LV is up.
Once ASIC LV is up and ASICs are reconfigured, check the status of the ASICs by taking a run and checking the plot below with the OM:
The first run number (18559 above) is the new run you just took. The second (17524) is the reference, preferably a run taken right before the power-down. If you see substantial differences in a crate associated to a full card (32 or 64) channels try first power-cycling the crate (with RO expert approval) before attributing the difference to a mis-configuration.