Project

General

Profile

Power Outage Planning » History » Version 22

Sophie Berkman, 07/29/2019 05:05 PM

1 1 David Caratelli
h1. Power Outage Planning
2 1 David Caratelli
3 1 David Caratelli
h2. Prep work:
4 1 David Caratelli
5 3 David Caratelli
* System experts need passwords: 
6 3 David Caratelli
> # PMT: Slow Controls HV power-supply control.
7 3 David Caratelli
> # TPC: glassman connection.
8 1 David Caratelli
9 1 David Caratelli
* We need a DAQ configuration that configures the ASICs. Examples from the past is config 768. Take data for any amount of time. Once run starts ASICs are configured. Make sure run does not re-start or you will re-configure the ASICs.
10 1 David Caratelli
11 13 David Caratelli
* We need a DAQ config to calibrate the PMTs. Current config is 792. Make sure this is available before the outage starts.
12 13 David Caratelli
13 18 David Caratelli
* Get in touch with Zarko to check that he will take care of powering off/on MiniBooNE.
14 18 David Caratelli
15 1 David Caratelli
* If you want to take data during ramp-up or ramp-down and the PMTs are off you will need a config which has an unbiased trigger and goes to the TestRun folder.
16 9 Pawel Guzowski
** If it is not in the TestRun folder, it will cause PUBS errors.
17 1 David Caratelli
18 2 David Caratelli
A power-outage schedule to guide planning can be found below. Outages generally start early in the morning, so this guide assumes the power-down procedure will take place the day before. In the plan below
19 2 David Caratelli
20 17 David Caratelli
h2. Guidelines:
21 2 David Caratelli
22 2 David Caratelli
* The cryo system should be down for as little time as possible to prevent buildup of impurities and a degradation of our electron lifetime. 
23 2 David Caratelli
24 14 David Caratelli
* The ASIC LV should be powered up as soon as this is possible. The ASIC LV powers the diodes which protect our electronics, hence the urgency. Likewise, ASIC LV should be one of the last items to be turned off. This should happen after other TPC HV systems are off, and after the PMT HV is off.
25 2 David Caratelli
26 20 David Caratelli
* Before leaving for the day the night before the outage, change the Run Plan instructing shifters to complete the "Power Outage Checklist" rather then the normal one, a streamlined version which forces the shifter to focus on subsystems / items which can still be monitored.
27 19 David Caratelli
28 4 David Caratelli
* Rack power-up/down instructions are on the [[Rack Power ONOFF procedures]] page.
29 4 David Caratelli
30 2 David Caratelli
* Subsystem dependencies: 
31 1 David Caratelli
> # TPC HV, ASIC LV, and wire-bias HV should come up only after the Cryo system is on. 
32 14 David Caratelli
> # ASIC LV should be powered down after PMT HV, wire-bias, and TPC HV are powered down.
33 3 David Caratelli
> # Data taking in BeamOff or Physics mode requires PMTs to be on. This is because otherwise PUBS will not be able to process incoming files.
34 15 David Caratelli
> # If turning on the laser rack, Slow Controls needs to be on before turning the laser key, otherwise the PMT interlock alarm will prevent the laser from turning on.
35 1 David Caratelli
> # PMT HV, Wire-bias, TPC HV all need the cryo system back up before being able to ramp back up.
36 17 David Caratelli
37 17 David Caratelli
h2. Step-by-step Guide:
38 4 David Caratelli
39 4 David Caratelli
Day before outage:
40 4 David Caratelli
41 4 David Caratelli
# 14:25 Stop data-taking - shifter (5 min.)
42 4 David Caratelli
# 14:30 Turn off PUBS - DM expert (30 min.)
43 4 David Caratelli
# 14:30 Turn off CRT remotely - CRT expert (30 min.)
44 22 Sophie Berkman
# 14:30 power down PMT HV - PMT expert (30 min.)
45 4 David Caratelli
# 15:00 ramp down TPC HV - TPC/HV expert (2-3 hours)
46 4 David Caratelli
# 17:00 ramp down wire-bias - TPC/HV expert (1 hour)
47 4 David Caratelli
# 18:00 power off ASIC LV - TPC/HV expert (30 min.)
48 4 David Caratelli
# 18:30 power down racks - RunCo + RO expert. CRT/MuCS expert also may be necessary. (1 hour.)
49 8 David Caratelli
 a Make sure data-taking has stopped before powering off any TPC/PMT rack.
50 8 David Caratelli
 b Make sure PMTs are powered off before touching the PMT-Trig crate. Check explicitly with PMT expert via phone.
51 8 David Caratelli
 c Make sure the TPC wire-bias and ASIC LV are powered off before powering down the TPC-PS rack. Check explicitly with the TPC-HV expert via phone.
52 8 David Caratelli
 d Make sure the TPC HV is powered down before turning off the TPC HV racks. Check explicitly with the TPC-HV expert via phone.
53 4 David Caratelli
54 4 David Caratelli
Day of outage, before:
55 4 David Caratelli
56 21 Glenn Horton-Smith
# T-30 min. "Turn off slow-controls":/redmine/projects/uboone-operations/wiki/SLC_-_Experts_Only#StoppingStarting-slow-controls-processes-beforeafter-power-outage. - SMC expert (10 min.)
57 4 David Caratelli
# T-20 min. Turn off non-evb machines - SLAM team (5 min.)
58 4 David Caratelli
# T-10 min. Turn off evb - SLAM team (5 min.)
59 4 David Caratelli
# T-5 min. Power down cryo - Cryo engineers, remotely (5 min.)
60 4 David Caratelli
61 4 David Caratelli
Day of / after outage, power-up:
62 4 David Caratelli
63 12 David Caratelli
# T+5 min. Power up cryo - Cryo engineers (up to 1 hour)
64 8 David Caratelli
 a. Make sure Keythleys in TPC-HV rack are configured. See e-log 77738: http://dbweb5.fnal.gov:8080/ECL/uboone/E/show?e=77738.
65 4 David Caratelli
# T+5 min. Power-up racks in LArTF - RunCo + RO expert (1 hour)
66 1 David Caratelli
# T+5 min. Power-on machines @ LArTF - SLAM team (5 min.)
67 21 Glenn Horton-Smith
# T+15 min. "Power-on slow controls":/redmine/projects/uboone-operations/wiki/SLC_-_Experts_Only#StoppingStarting-slow-controls-processes-beforeafter-power-outage - Slow Controls (30 min.)
68 12 David Caratelli
# T+1 hour. Turn on ASIC LV. - TPC/HV expert (30 min.)
69 12 David Caratelli
# T+1 hour. Turn on PUBS - DM expert (30 min.)
70 1 David Caratelli
# T+1 hour Turn on CRT - CRT expert (30 min.)
71 22 Sophie Berkman
# T+1 hour Turn on PMTs - PMT expert (30 minutes)
72 12 David Caratelli
# T+1:30 hour Turn on wire-bias HV - TPC/HV expert (1 hour)
73 1 David Caratelli
# T+2:30 hours Ramp up TPC HV - TPC/HV expert (2/3 hours)
74 1 David Caratelli
# Start taking data. First run should use config ID that re-configures the ASICs. This step can be performed once the ASIC LV is up.
75 22 Sophie Berkman
# PMT Calibration run (30 minutes)
76 22 Sophie Berkman
# Start taking a normal data run
77 16 David Caratelli
78 16 David Caratelli
Once ASIC LV is up and ASICs are reconfigured, check the status of the ASICs by taking a run and checking the plot below with the OM:
79 16 David Caratelli
http://ubdaq-prod-near2.fnal.gov/lizard/18559/17524/#/tpc/mapccc/h_pedestal_width
80 16 David Caratelli
The first run number (18559 above) is the new run you just took. The second (17524) is the reference, preferably a run taken right before the power-down. If you see substantial differences in a crate associated to a full card (32 or 64) channels try first power-cycling the crate (with RO expert approval) before attributing the difference to a mis-configuration.