|Shifter||1 (937) 582-6663||1(937) 5-UBOONE|
|Backup Shifter||1 (978) 822-2582||9788 BACK UB|
|ON-CALL: RunCo (Ralitsa)||(617) 230-2978|
|ON-CALL: Deputy RunCo (Avinay)||(315) 350-9065|
|ON-CALL: Deputy RunCo (Giacomo)||(203) 435-0293|
- Currently the slowmon box on CRT DAQ rack 1 cannot be reached. This causes INVALID (pink) alarms on RackProt/System CRT_1_0 and RackTemps/System CRT_1_1. Experts are aware of the issue and working to solve it, no need to slack, email or call. Please note that RackProt and RackTemps alarms on any system other than CRT_1_x still need to be reported!
- If you notice at any time the pink X (beam position) on the BNB status page is not being displayed:
- if you know how to use the ACNET console, please plot E:HP875 and E:VP875 logged on 1D. If you get "Sorry, no data" error please call MCR and tell them those devices are not being logged on MBoon 1D 40 mSec.
- if you do not know how to use the ACNET console, please call the RunCo (even if it is in the middle of the night).
- Remote shifters please note that IFIX is working fine. If you were not able to connect, talk to your PI and work on upgrading your RDP and/or OS immediately. IFIX is a requirement for having a certified remote shift station!
- uB_ZMON_ZMON_1/short_ok is firing occasionally. Experts are aware. No need to call, email/slack anyone. Note this is only for this particular ZMON alarm, any other ZMON alarm should be reported.
- H2O monitor uB_Cryo_IFIX_1_0/AT608 sometimes rises due to outside dewar being filled or due to pressure variations. No need to call, email/slack runco if in alarm.
- If you see alarms for beam position variables HPTG1 or VPTG1 you can ignore these; those are not used by AD to steer the beam currently.
- uB_OnDetPower_TPCPS_1_5_404/CURR_READ alarm is acknowledged. Experts are aware.
- If you see a uB_DAQStatus_DAQX/sn_read_lag_multiplicity alarm:
- Check whether it is an old alarm that is still latched. If it has a green OK status box you can acknowledge it.
- Check whether the run is crashing/restarting. This alarm is frequently triggered by the run stopping (for any reason). It is generally not the cause of a run crash.
- If the alarm persists for a few minutes without the run stopping/restarting:
- If it's a minor alarm, elog which SEB is having the sn_read_lag alarm and add the readout expert to the elog entry recipient list.
- If it's a major alarm, restart the run once and elog it as above. If the major alarm returns, then call the readout expert.
- (For more details see: https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/RC_-_Troubleshooting#Im-getting-a-MAJOR-alarm-on-uB_DAQStatus_DAQXsn_read_lag_multiplicity-What-should-I-do)
- If you are local and you notice the yellow computer slowing down (cr-02):
1. Check the "about:performance" tab in the browser. (If the tab is not open, just type "about:performance" in the address line of a new tab). See (and make note of) if any of the web apps are performing poorly.
2. Close and restart any pages that are performing poorly, or just close and reopen all Firefox windows.
3. Email or Slack message (don't call) the run coordinator and tell them which app was causing the slow down.
|Configuration||Config ID||Time [min]|
Notes on current operations¶
- Fill in the usual shift checklist during the shift 2 hours in and 2 hours before end of shift.
- General procedure: some troubleshooting instructions advocate to stop and restart the run to fix a problem. It is good to do so If stopping and restarting didn't work at the first try, please get in touch with the relevant experts (or the Run Co) and
- Before resetting the PUBS, please ensure there are no PUBS errors being displayed - if there are, please report this to the Data Management expert (via email) before resetting.
- If you make an elog entry about online monitor or runcat checks failing, upload a screenshot of the plot with the entry.
Additional Shifter Requirements:¶
- Please inform the Run Coordinator(s) if
- you see any DAQ instabilities, or
- the mean run duration gets smaller than 2h, or
- you receive any information from MCR about the beam, or
- there is any mismatch between beam triggers and beam spills (dots and lines of the same color in the nearline trigger monitor).
- Please do not forget to check your runs in the RunCat.
- Please fill out the Offline Production Checklist at 9am and 9pm every day.
There are instruction on how to fill this out on this page: https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Offline_Production_Checklist_-_Shifters
- Please note that the offline production checklist has been modified. The content is essentially the same, but if you've done it before, it will look different now. Also, now you email the checklist itself to the offline production listserv rather than composing a separate email.
- Please remember to fill in the CI validation check-list once per shift block (on Mondays and Fridays)
There are instructions on how to check the plots on this page: https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/CI_Validation_Instructions
- Please follow the Shift_turnover procedures
- During access to the detector:
- Make sure the Run Co's were informed of the access (Slack chat or call).
- Detector platform access must be cleared by Run Co's.
- Please fill out a Beginning of Detector Access and End of Detector Access Form.
- If the "keytree" alarm occurs, and the persons accessing the platform have not called or do not call the shifter within ~5 minutes, then call LArTF to find out who is accessing the detector. If no one answers, call the runco.
- Please remind to the people making the access to call you back when they are done with their work in the pit.
- At the end of detector access, please acknowledge the keybank incomplete alarm.
- If you happen to see a pretty event in the event display, take a screenshot!
What should I do if I see alarms during my shift?¶
Have a look at this wiki page: What to do in case of alarms during a shift
Expert contact information: https://microboone-exp.fnal.gov/at_work/expert_call_list.html