Run Plan - last updated Mar 24th 12:04 am CDT¶
|Shifter||1 (937) 582-6663||1(937) 5-UBOONE|
|Backup Shifter||1 (978) 822-2582||9788 BACK UB|
|ON-CALL: RunCo (Adrien)||630 544 9094||(call 2x within 3min if I don't answer at first)|
|OFF-CALL: Deputy RunCo (Simone)||773 895 6815||(don't be shy and call multiple times if needed)|
Summary¶Please inform the Run Coordinator(s) if
- you see any DAQ instabilities, or
- the mean run duration gets smaller than 2h, or
- if you receive any information from MCR about the beam.
Minor mem_free alarms are OK at the moment you can ignore them for now
Please do not forget to check your runs in the RunCat.
Please remember to fill in the CI validation check-list once per shift block (on Mondays and Fridays)
There are instructions on how to check the plots on this page: https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/CI_Validation_Instructions
|Configuration||Config ID||Time [min]|
Notes on current operations:¶
- The sn_read_lag error is currently our most frequent DAQ alarm. This alarm sometimes appears as a symptom of runs ending. If it clears when the new run begins there is no cause for concern. If you see the alarm appear during an ongoing run, follow the instructions on the Run Control Troubleshooting page. We are trying to monitor its frequency.
- We've been seeing frequent disk_free/mem_free errors on seb06. If the error is minor and stays up for at least 15-20minutes, then email the Data Management expert (this is NOT a DAQ issue). In case it is major, please get in touch immediately with the on-call Data Management expert. Anyway, restarting a run is NOT the way to go for this error.
- Before resetting the PUBS, please ensure there are no PUBS errors being displayed - if there are, please report this to the Data Management expert (via email) before resetting.
- Passwords can be found on http://dbweb6.fnal.gov:8080/ECL/uboone/PL/project?pid=2 if the password you look for is not here, please add it (once you figure it out) or ask the run coordinator to do it
Additional Shifter Requirements:¶
- Please follow the Shift_turnover procedures
- At the end of your shift, please post a plot of the last 8 hours of variable: uB_Cryo_IFIX_1_0/PT102. If in this plot you see oscillations/spike that concern you, please read the next section (what to do in case of alarms - cryo alarm section)
- During access to the detector:
- Make sure the Run Co's were informed of the access (google chat or call).
- Detector platform access must be cleared by Run Co's.
- Please fill out a Beginning of Detector Access and End of Detector Access Form.
- If the "keytree" alarm occurs, and the persons accessing the platform have not called the shifter first, then call LArTF to find out who is accessing the detector. If no one answers, call the runco.
- Please remind to the people making the access to call you back when they are done with their work in the pit.
- At the end of detector access, please acknowledge the keybank incomplete alarm.
What should I do if I see alarms during my shift?¶
If you see a MAJOR alarm in the Slow Controls GUI:
- If BeamData: check MCR elog for explanation (https://www-bd.fnal.gov/Elog/), contact Run Coordinator.
- If the BNB status page has plots in alarm (colored in yellow or red) for more than 30minutes, please contact the Run Coordinator.
- If Cryo: check for corresponding IFIX alarms. See the instructions below for what to do if you see a major Cryo alarm.
- If you are getting a minor alarm for the variable uB_Cryo_IFIX_1_0/PT102: be aware that minor oscillations below 0.1psig are indicative of the LN2 dewar refilling and this is normal. Additionally a small spike every 7 hours is expected. Example: http://dbweb6.fnal.gov:8080/ECL/uboone/E/image/47025/Screenshot%20from%202017-08-14%2014-57-26.png
- If DriftHV or OnDetPower: contact on-call TPC/HV expert immediately.
- If PMTHV: contact on-call PMT expert immediately.
- If CRT Crate Rails alarms (uB_CrateRails_CRTUtil*): Minor Alarm - keep an eye to that variable and send an email to the CRT on-call expert at the end of the shift with the last 8h plot. Major Alarm - contact immediately the CRT on-call expert.
- If CrateRails or FEM/XMIT: contact on-call Readout Electronics expert immediately. For FEM/XMIT alarms that oscillate and do not remain for any sizeable portion of the run, you can email the expert.
- If sn_read_lag: this alarm will appear as a symptom of runs ending; if it clears when the new run begins there is no cause for concern. If you see the alarm appear during an ongoing run, follow the instructions on the Shifter Troubleshooting Page - near the bottom).
- If DAQStatus (except for FEM/XMIT or sn_read_lag): check the Shifter Troubleshooting Page for guidance. If you have any troubles, can't find the error you see, or if instructed to do so by the troubleshooting page, contact on-call DAQ expert immediately (try Google Chat first but don't wait too long for a response)
- If PCStatus: for alarms not related to disk space/loads, contact DAQ expert immediately. For alarms related to disk space/loads, contact Data Management expert immediately.
- If RackFans, RackProt or RackTemps: contact run coordinator immediately.
- If ZMON: contact run coordinator immediately, inform anyone on platform at LArTF to stop work and ask them to contact run coordinator also.
If you see a MINOR alarm in the Slow Controls GUI, please contact the expert as specified above, but email and wait for a response rather than calling (particularly at unsociable hours). You can also try the Google Chat.
If you have any other problems not described above, check the Shifter Troubleshooting Page for guidance. If that doesn't help, try Google Chat. If it's a serious problem please don't wait for a response - contact the Run Co-ordinator or most relevant expert immediately!
In Case of Cryo Alarms¶
All the cryo related alarms from either iFix or slow controls should be documented by shifters in ECL with the alarm variable names.If a shifter does not receive a call within 15 minutes of seeing a MAJOR cryo alarm:
- Call the Run Coodinator
- If the Run Coordinator cannot be reached
- During Daytime Operating Hours (8am - 10pm CDT)
- Call the cryo experts in the order listed on the expert call list: http://www-microboone.fnal.gov/at_work/expert_call_list.html
- During nighttime hours:
- Email the cryo experts
- During Daytime Operating Hours (8am - 10pm CDT)