What to do in case of alarms during a shift¶
If you see an INVALID alarm, contact the Slow Controls expert.
If you see a MAJOR alarm in the Slow Controls GUI:
- If the variable is related to the age of a system's data (e.g.,
uB_TPCDrift_HV01_1_0/age, etc.), then contact the Slow Controls Expert.
- If BeamData: check MCR elog for explanation (https://www-bd.fnal.gov/Elog/), contact Run Coordinator.
- If the BNB status page has plots in alarm (colored in yellow or red) for more than 30minutes, please contact the Run Coordinator.
- If Cryo:
- In case of minor
Keybank_Incompletealarm, see the Run Plan procedure for "During access to the detector". At the end of detector access, the Current Status of the alarm in the Alarm Table should change to "OK", but the alarm will remain. Please acknowledge the keybank incomplete alarm once its current status is "OK".
- If you are getting a minor alarm for the variable uB_Cryo_IFIX_1_0/PT102: be aware that minor oscillations below 0.1psig are indicative of the LN2 dewar refilling and this is normal. Additionally a small spike every 7 hours is expected. Example: http://dbweb6.fnal.gov:8080/ECL/uboone/E/image/47025/Screenshot%20from%202017-08-14%2014-57-26.png
- For Major alarms, check for corresponding IFIX alarms. Check the Run Plan for any special instructions. Contact the run coordinator if no other instruction is available.
- In case of minor
- If TPCDrift (drift HV) or OnDetPower: contact on-call TPC/HV expert immediately unless you get a "blip" alarm. (Blip alarms appear as "Blip monitor
uB_TPCDrift_HV01_keithleyPickOff/voltDiff5s60s".) In case of a "blip" error, make an elog entry cc'ing the HV expert and the Run Co. Acknowledge the blip alarm after the current status changes to "OK" in the Alarm Table. If you get more than 1 blip in your shift, call the TPC/HV expert immediately.
- If PMTHV: contact on-call PMT expert immediately.
- If CRT Crate Rails alarms (uB_CrateRails_CRTUtil*): Minor Alarm - keep an eye to that variable and send an email to the CRT on-call expert at the end of the shift with the last 8h plot. Major Alarm - contact immediately the CRT on-call expert.
- If CrateRails : contact the on-call Readout Electronics expert via phone. If the alarms oscillate and do not remain on for any sizable time, an email is sufficient.
- If xmit_frame_ctr_diff or xmit_trigger_ctr_diff alarms appear: follow the instructions available at the Shifter Troubleshooting Page (third entry from bottom of the page).
- If sn_read_lag_multiplicity: this alarm will appear as a symptom of runs ending; if it clears when the new run begins there is no cause for concern. If you see the alarm appear during an ongoing run, follow the instructions on the Shifter Troubleshooting Page (second entry from the bottom of the page).
- If DAQStatus (except for FEM/XMIT or sn_read_lag_multiplicity): check the Shifter Troubleshooting Page for guidance. If you have any troubles, can't find the error you see, or if instructed to do so by the troubleshooting page, contact on-call DAQ expert immediately (try Slack Chat first but don't wait too long for a response).
- If PCStatus: for alarms not related to disk space/load_one, contact the DAQ expert. For alarms related to disk space, contact Data Management expert immediately. For alarms related to loads (e.g. load_one on some machine) contact the Data Management expert immediately only if the alarms are persistent and last more than a few minutes. Otherwise, just log the happening in the ECL.
- If RackFans, RackProt or RackTemps: contact run coordinator immediately.
- If ZMON: contact run coordinator immediately, inform anyone on platform at LArTF to stop work and ask them to contact run coordinator also.
If you see a MINOR alarm in the Slow Controls GUI, please contact the expert as specified above, but email and wait for a response rather than calling (particularly at unsociable hours). You can also try the Slack Chat.
- if you see a big red box in the Online Monitor Lizard: click on "Start here" at the bottom of the red box and look at what variable is in alarm. Get in touch with the Run Co.
- If PUBS is causing alarms, contact the Data Management expert.
- If sn_read_lag or sn_read_lag_multiplicity: for a MINOR alarm, add an elog entry with which SEB is having the sn_read_lag alarm and add the readout expert as recipient of the elog entry.
If you have any other problems not described above, check the Shifter Troubleshooting Page for guidance. If that doesn't help, try Slack Chat. If it's a serious problem please don't wait for a response - contact the Run Co-ordinator or most relevant expert immediately!
Expert contact information: https://microboone-exp.fnal.gov/at_work/expert_call_list.html