Wiki » History » Version 272

« Previous - Version 272/370 (diff) - Next » - Current version
Michael Mooney, 01/27/2016 06:52 PM

Before you start your shift

  • Make sure you have a current Fermilab ID, Fermilab Kerberos account, and Fermilab Services account (and go to "Request New/Renew Accounts" or call 630-840-2345 - 24 hours a day).
  • Ask the Run Coordinator to add your Kerberos principal to the .k5login files for the uboonedaq and ubooneshift users on uboone-cr-01, uboone-cr-02, uboone-cr-03, and uboone-cr04.

Shift Procedure

Review Operations Wiki Updates page

  • Changes in shift procedure and important notices for shifters are listed by date in the Operations Wiki Updates page. Please review this list at the beginning of every shift and contact the relevant subsystem expert or Run Coordinator if you are not clear on new procedures.
  • Be sure to reread parts of the Operations Wiki that have been updated (or are new) since the last shift you took. The Operations Wiki Updates page should make it very simple to determine which material should be reviewed.
  • If your last shift was before October 14th, 2015 (the date the Operations Wiki Update page was created), please read the entire Operations Wiki. Ideally this should be done before the beginning of your shift.

Log on to the control room computers

  • On the mac, the user is ubooneshift, and the password is the same as the MicroBooNE DocDB password.
  • On the Linux boxes, there are two cases (You must be added to the .k5login file. Contact us to help):
    • Case 1. The screen has been locked. In this case enter your Kerberos password, followed by your Keberos principal (user name).
    • Case 2. You have been logged out. In this case first enter the user as "ubooneshift", then enter your Kerberos password, and finally enter your Keberos principal (user name).

REFRESH ALL OPEN BROWSERS and sign in to E-log, Google+

  • Browsers open with instructions may have been changed! Make sure you have the latest instructions.

Check for special run plans, else follow the routine shift procedure

  • At the beginning of the shift call MCR (x3721) and let them know that the new MicroBooNE shifters are in ROC West.
  • At the beginning of shift, fill out the Beginning of Shift form on the E-log, and enter it under topic "Shift/Beginning of shift".
  • During shift:
    • Take runs according to the Run Plan, referencing RC - Guide for instructions about the run control. Contact DAQ on-call expert if there are any problems in data-taking (first following the recommendations in RC - Guide) - try Google Chat first and give the expert a phone call if they don't respond within a minute.
    • Every four hours (two hours into the shift and six hours into the shift), fill out the Checklist form on the E-log, and enter it under topic "Shift/Checklist".
      • This checklist is used to ensure shifters are focusing on the right things during their shifts, to help track our noise issues as they develop over time, to catch things that "slip between the cracks" of our slow monitoring alarms, and to provide a summary of who to contact in case problems come up.
    • Monitor the BNB status display (see Beam - Overview for instructions) and follow the directions in Beam - Guide about alarms or significant changes in beam conditions.
    • Occasionally (every hour or so) check the MCR E-log for updates about the status of the BNB.
    • Monitor the slow control display for recent alarms (see SLC - Guide). Check for alarming/tripping power supplies and call the relevant subsystem expert if you see major alarms. Acknowledge previously understood alarms, and new major alarms after contacting the relevant subsystem expert. In either case fill out the Alarm Acknowledged form on the E-log. For minor alarms, send an email to the relevant subsystem expert instead.
      • For cryo-related alarms in the slow control display, before contacting anyone you should first verify that the cryo IFIX display is also showing an alarm. The cryo IFIX display is located on the opposite side of the NOvA shifter station (by the white boards along the side of the room). The machine with the IFIX display is the one closest to the door (but not the one facing the door that is used to control the displays on the walls). The username is "Nova" and the password is "2The" followed by a word describing what the night sky is filled with (first letter capitalized) followed by a symbol for US currency. If the IFIX display is not showing an alarm as well, contact the Run Coordinator (phone call for major alarm, email for minor alarm) instead of MC-1 (as instructed in SLC - Guide).
    • Look at the Online Monitor plots for the runs you take while on shift. See an example of things to monitor here: OM - Checklist.
    • Monitor data processing, a.k.a. PUBS. Make sure PUBS is running and that the computers on which PUBS is running are not overloaded. See the specific instructions at DM - Shifters.
    • Please promptly update the Run Information Document with run information as the runs are taken during shift.
      • This document includes ASIC configuration settings as well as information about activity at LArTF during the collection of data. Please continue to update this document during early operations as it will be very helpful in helping us diagnose noise-related concerns, for instance. Please update this document during your shift and copy your updates into an E-log entry at the end of your shift.
      • Also note that noting operations-related items in the Run Information Document or making a note in the shifter Google chat window is not a replacement for an E-log entry. Any operations-related items of interest must be noted in the E-log, both promptly and with specific information (such as which alarms went into a bad state).
    • Shifters need to notify Run Coordinator when experts are going to work in LArTF and also when the shifters are asked to help the experts in LArTF.
      • Shifters must also fill out an E-log entry using the Start of Detector Access form, and fill out the End of Detector Access form when the work is complete.
      • Be sure to follow the LArTF access procedures for shifts.
  • At the end of shift, fill out the End of Shift form on the E-log, and enter it under topic "Shift/End of shift". Communicate with the incoming shifters (who should arrive at least 10 minutes before the end of the shift) the events of the shift, anything unusual they should be monitoring, and changes in the Run Plan communicated to you by the Run Coordinator or DAQ experts.

Shift Workstations

  • Workstation Layout: this page shows what should be where, on the workstation monitors in front of you

Run Control

Slow Controls/Monitoring

  • SLC - Guide: What should I look at? How should it look like?

Online Monitoring Lizard

  • OM - Errors A list of errors that can come up with the Big Red Box

Data Management and PUBS

Laser System

Light Collection/PMT System

Beam Monitoring

Raspberry Pi Webcam

Cathode High Voltage

Readout Electronics

Time Projection Chamber (TPC)

  • TPC-Overview: This is a shifter guide to what are the relevant systems associated with the TPC
  • TPC-Guide: Here is a step-by-step guide to monitoring things related to the operation of the TPC
  • TPC-Troubleshooting: Quick guide to previously experienced problems with the TPC (When in doubt call the TPC expert)

Purity Monitors


Online Event Display (Mac Event Display)

Off-Site Shifts

Example System

  • Example - Overview: This is basically what should the shifter be looking at (not necessarily touching), try to be explicit about what give what information about what to the shifter.
  • Example - Guide: How should the shifter navigate through the different screens to retrieve the necessary information
  • Example - Experts Only: This is where the "Expert Documentation" should live, but only for the time being, we will want to move this behind a password wall. For the time being please start to build the documentation here now but be mindful that it is public viewable.

Call Lists

Expert call list (password-protected)

Collaboration contact list (password-protected)

Useful Links

Expert actions

  • k5login Adding shifters to the .k5login
  • Keytab: Automatically renew the kerbero tickets on evb and control room machines.
  • MicroBooNE spares: Follow this link to see a list of MicroBooNE spares, location, and date last checked.