Project

General

Profile

Run plan » History » Version 661

« Previous - Version 661/1543 (diff) - Next » - Current version
Adrien Hourlier, 03/30/2018 10:27 AM


Run Plan - last updated Mar 30th 11:30 am CDT

Contacts

Contact Number Alt Number
Shifter 1 (937) 582-6663 1(937) 5-UBOONE
Backup Shifter 1 (978) 822-2582 9788 BACK UB
ON-CALL: RunCo (Simone) 773 895 6815 (don't be shy and call multiple times if needed)
OFF-CALL: Deputy RunCo (David C.) 847 275 9484

Link to which experts to call in case of other alarms

Run Configuration

Summary

Please inform the Run Coordinator(s) if
  • you see any DAQ instabilities, or
  • the mean run duration gets smaller than 2h, or
  • if you receive any information from MCR about the beam.

Please do not forget to check your runs in the RunCat.

Please remember to fill in the CI validation check-list once per shift block (on Mondays and Fridays)
There are instructions on how to check the plots on this page: https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/CI_Validation_Instructions

Run Control

Configuration Config ID Time [min]
PHYSICS_BNB_NUMI_MUCS_EXT15990mHz_SWTrigger_2Stream_LowThreshold50_configurableQueuesNUSNstreamEVB3000SEB500TRG200_circularbuffer2G.fcl 728 420

Notes on current operations:

  • The sn_read_lag error is currently our most frequent DAQ alarm. This alarm sometimes appears as a symptom of runs ending. If it clears when the new run begins there is no cause for concern. If you see the alarm appear during an ongoing run, follow the instructions on the Run Control Troubleshooting page. We are trying to monitor its frequency.
  • We've been seeing frequent disk_free errors on seb06. If the error is minor and stays up for at least 15-20minutes, then email the Data Management expert (this is NOT a DAQ issue). In case it is major, please get in touch immediately with the on-call Data Management expert. Anyway, restarting a run is NOT the way to go for this error.
  • If you get mem_free errors on the seb's or evb, please log that in the ECL. Minor alarms are OK for now. If the errors get major, get in touch with the DAQ expert. However, wait for their instruction to stop/restart the run.
  • Before resetting the PUBS, please ensure there are no PUBS errors being displayed - if there are, please report this to the Data Management expert (via email) before resetting.

Additional Shifter Requirements:

  • Please follow the Shift_turnover procedures
  • At the end of your shift, please post a plot of the last 8 hours of variable: uB_Cryo_IFIX_1_0/PT102. If in this plot you see oscillations/spike that concern you, please read the next section (what to do in case of alarms - cryo alarm section)
  • During access to the detector:
    • Make sure the Run Co's were informed of the access (google chat or call).
    • Detector platform access must be cleared by Run Co's.
    • Please fill out a Beginning of Detector Access and End of Detector Access Form.
    • If the "keytree" alarm occurs, and the persons accessing the platform have not called the shifter first, then call LArTF to find out who is accessing the detector. If no one answers, call the runco.
    • Please remind to the people making the access to call you back when they are done with their work in the pit.
    • At the end of detector access, please acknowledge the keybank incomplete alarm.

What should I do if I see alarms during my shift?

If you see a MAJOR alarm in the Slow Controls GUI:

  • If BeamData: check MCR elog for explanation (https://www-bd.fnal.gov/Elog/), contact Run Coordinator.
  • If the BNB status page has plots in alarm (colored in yellow or red) for more than 30minutes, please contact the Run Coordinator.
  • If Cryo: check for corresponding IFIX alarms. See the instructions below for what to do if you see a major Cryo alarm.
  • If DriftHV or OnDetPower: contact on-call TPC/HV expert immediately.
  • If PMTHV: contact on-call PMT expert immediately.
  • If CRT Crate Rails alarms (uB_CrateRails_CRTUtil*): Minor Alarm - keep an eye to that variable and send an email to the CRT on-call expert at the end of the shift with the last 8h plot. Major Alarm - contact immediately the CRT on-call expert.
  • If CrateRails or FEM/XMIT: contact on-call Readout Electronics expert immediately. For FEM/XMIT alarms that oscillate and do not remain for any sizeable portion of the run, you can email the expert.
  • If sn_read_lag: this alarm will appear as a symptom of runs ending; if it clears when the new run begins there is no cause for concern. If you see the alarm appear during an ongoing run, follow the instructions on the Shifter Troubleshooting Page - near the bottom).
  • If DAQStatus (except for FEM/XMIT or sn_read_lag): check the Shifter Troubleshooting Page for guidance. If you have any troubles, can't find the error you see, or if instructed to do so by the troubleshooting page, contact on-call DAQ expert immediately (try Google Chat first but don't wait too long for a response)
  • If PCStatus: for alarms not related to disk space/loads, contact DAQ expert immediately. For alarms related to disk space/loads, contact Data Management expert immediately.
  • If RackFans, RackProt or RackTemps: contact run coordinator immediately.
  • If ZMON: contact run coordinator immediately, inform anyone on platform at LArTF to stop work and ask them to contact run coordinator also.

If you see a MINOR alarm in the Slow Controls GUI, please contact the expert as specified above, but email and wait for a response rather than calling (particularly at unsociable hours). You can also try the Google Chat.

If you have any other problems not described above, check the Shifter Troubleshooting Page for guidance. If that doesn't help, try Google Chat. If it's a serious problem please don't wait for a response - contact the Run Co-ordinator or most relevant expert immediately!

In Case of Cryo Alarms

All the cryo related alarms from either iFix or slow controls should be documented by shifters in ECL with the alarm variable names.

If a shifter does not receive a call within 15 minutes of seeing a MAJOR cryo alarm: