Project

General

Profile

Read the Run Plan

  • Refresh and read the Run Plan at the start of every 8 hour block of shifts.

Before you start your shift

  • Arrange for training with the Run Coordinator
    • This requires 1 hour of in-person training with a current or previous run coordinator. Group training sessions will be organised periodically that are accessible both locally at FNAL and remotely, although remote attendees will require access to the shared Slow Controls GUI and a separate Zoom screen-share. Contact the run coordinator to be put on the list for the next session. Other training sessions can be arranged in person with the run coordinators when available at FNAL; remote training is not available outside the scheduled sessions.
    • After the initial training, a new shift is required to take 2 shadow shifts. Please have the shadowed shifters email the Run Coordinator confirm that the shadow shifts are complete.
    • Ask the Run Coordinator or other expert to add your Kerberos principal to the MicroBooNE .k5login

Shift Procedure

Review Operations Wiki Updates page

  • Changes in shift procedure and important notices for shifters are listed by date in the Operations Wiki Updates page. Please review this list at the beginning of every shift and contact the relevant subsystem expert or Run Coordinator if you are not clear on new procedures. Last change: April 18, 2018.

Follow The Shift Turnover Procedures

  • Please note that you are required to check in to your shift ten minutes before the official start of your shift, as listed on the elog. Otherwise, follow these procedures for communicating with the outgoing shifter: Shift Turnover Procedures

Log on to the control room computers

  • On the mac, the user is ubooneshift, and the password is obtainable from the previous shifter or from the RunCo.
  • On the Linux boxes, there are two cases (You must be added to the .k5login file. Contact us to help):
     
    • Case 1. The screen has been locked. In this case enter your Kerberos password, followed by your Keberos principal (user name).
       
    • Case 2. You have been logged out. In this case first enter the user as "ubooneshift", then enter your Kerberos password, and finally enter your Keberos principal (user name).

REFRESH ALL OPEN BROWSERS and sign in to E-log, Google+

  • Browsers open with instructions may have been changed! Make sure you have the latest instructions.
  • SIGN INTO E-log WITH YOUR ACCOUNT: http://dbweb0.fnal.gov/ECL/uboone/E/index
    All shift-related E-log entries should use the "Shift" category.
    • Proper use of the e-log: making a note in the shifter Google chat window is not a replacement for an E-log entry. Any operations-related items of interest must be noted in the E-log, both promptly and with specific information (such as which alarms went into a bad state).

Check for special run plans, else follow the routine shift procedure

  • At the beginning of the shift call MCR (x3721) and let them know that the new MicroBooNE shifters are in ROC West.
  • At the beginning of shift, fill out the Beginning of Shift form on the E-log, and enter it under topic "Shift/Beginning of shift".
  • During shift:
     
    • Take runs according to the Run Plan, referencing RC - Guide for instructions about the run control. Contact DAQ on-call expert if there are any problems in data-taking (first following the recommendations in RC - Guide) - try Google Chat first and give the expert a phone call if they don't respond within a minute.
       
    • Every four hours (two hours into the shift and six hours into the shift), fill out the Checklist form on the E-log, and enter it under topic "Shift/Checklist".
       
      • This checklist is used to ensure shifters are focusing on the right things during their shifts, to help track our noise issues as they develop over time, to catch things that "slip between the cracks" of our slow monitoring alarms, and to provide a summary of who to contact in case problems come up.
    • Monitor the BNB status display (see Beam - Overview for instructions) and follow the directions in Beam - Guide about alarms or significant changes in beam conditions.
       
    • Occasionally (every hour or so) check the MCR E-log for updates about the status of the BNB.
       
    • Monitor the slow control display for recent alarms (see SLC - Guide). Check for alarming/tripping power supplies and call the relevant subsystem expert if you see major alarms. Acknowledge previously understood alarms, and new major alarms after contacting the relevant subsystem expert. In either case fill out the Alarm Acknowledged form on the E-log. For minor alarms, send an email to the relevant subsystem expert instead.
       
      • For cryo-related alarms in the slow control display, before contacting anyone you should first verify that the cryo IFIX display is also showing an alarm. The cryo IFIX display is located on the opposite side of the NOvA shifter station (by the white boards along the side of the room). The machine with the IFIX display is the one closest to the door (but not the one facing the door that is used to control the displays on the walls). Use the default username; the password is obtainable from the previous shifter or from the RunCo. If the IFIX display is not showing an alarm as well, contact the Run Coordinator (phone call for major alarm, email for minor alarm) instead of MC-1 (as instructed in SLC - Guide).
         
    • Look at the Online Monitor plots for the runs you take while on shift. See an example of things to monitor here: OM - Checklist.
       
    • Monitor data processing, a.k.a. PUBS. Make sure PUBS is running and that the computers on which PUBS is running are not overloaded. See the specific instructions at DM - Shifters.
       
    • Please check the quality of a data run using the Run Catalogue once a run finishes. Add in any comments about occurrences during the run.
    • This replaces the Run Information Document
       
  • Shifters need to notify Run Coordinator when experts are going to work in LArTF and also when the shifters are asked to help the experts in LArTF.
     
    • Shifters must also fill out an E-log entry using the Start of Detector Access form, and fill out the End of Detector Access form when the work is complete.
       
    • We have a DEDICATED form for the weekly cryo walk-through. You do NOT have to inform the RC about these.
       
    • Be sure to follow the LArTF access procedures for shifts.
  • At the end of shift, fill out the End of Shift form on the E-log, and enter it under topic "Shift/End of shift". Communicate with the incoming shifters (who should arrive at least 10 minutes before the end of the shift) the events of the shift, anything unusual they should be monitoring, and changes in the Run Plan communicated to you by the Run Coordinator or DAQ experts.

Shift Workstations

  • Workstation Layout: this page shows what should be where, on the workstation monitors in front of you

Run Control

  • RC - Experts Only: Old DAQ Expert Documentation (out of date - use the new DAQ expert documentation from the link above!)

Slow Controls/Monitoring

  • SLC - Guide: What should I look at? How should it look like?

Online Monitoring Lizard

  • OM - Errors A list of errors that can come up with the Big Red Box

Run Catalogue

  • Fill in the Run Catalogue as runs are taken, to check the quality of the data

Data Management, Production, and PUBS

Laser System

Light Collection/PMT System

Beam Monitoring

Raspberry Pi Webcam

Readout Electronics

Time Projection Chamber (TPC) and Drift High Voltage (HV)

  • TPC-Overview: This is a shifter guide to what are the relevant systems associated with the TPC
  • TPC-Guide: Here is a step-by-step guide to monitoring things related to the operation of the TPC
  • TPC-Troubleshooting: Quick guide to previously experienced problems with the TPC (When in doubt call the TPC/HV expert)
  • DEPRECATED. See TPC/HV Expert document in above link Cathode High Voltage - Troubleshooting: Quick guide to previously experienced problems related to TPC drift high voltage (When in doubt call the TPC/HV expert)

Cryo

Purity Monitors

Nearline Monitoring

MiniBooNE

Online Event Display

Argo Event Display

  • Argo evd - Guide: A brief overview of the Argo event display and what it is doing.

Off-Site Shifts

Muon Counter System (MuCS)

Cosmic Ray Tagger System (CRT)

Continuous Integration Validation

Example System

  • Example - Overview: This is basically what should the shifter be looking at (not necessarily touching), try to be explicit about what give what information about what to the shifter.
  • Example - Guide: How should the shifter navigate through the different screens to retrieve the necessary information
  • Example - Experts Only: This is where the "Expert Documentation" should live, but only for the time being, we will want to move this behind a password wall. For the time being please start to build the documentation here now but be mindful that it is public viewable.

Call Lists

Expert call list (password-protected)

Collaboration contact list (password-protected)

Useful Links

Expert actions

  • k5login Adding shifters to the .k5login
  • Keytab: Automatically renew the kerbero tickets on evb and control room machines.
  • MicroBooNE spares: Follow this link to see a list of MicroBooNE spares, location, and date last checked.
  • HV trip recovery: Actions and checks following an HV trip to ensure the detector is healthy before ramping the HV again.
  • Supernova protocol: Instructions for dealing with a supernova core-collapse event candidate (SNEWS alert) and the MicroBooNE SN stream (a.k.a. continuous readout).