Project

General

Profile

Wiki » History » Version 345

Version 344 (Simone Marcocci, 05/11/2018 12:45 PM) → Version 345/370 (Michael Weber, 06/13/2018 01:14 AM)

h1. Read the Run Plan

{{>toc}}

* Read the *%{background:white; color:blue}[[run_plan|Run Plan]]%* at the start of every 8 hour block of shifts.

h1. Before you start your shift

* Make sure you have:
** A current Fermilab ID
** A Fermilab Kerberos account
** A Fermilab Services account
** If not, https://fermi.service-now.com/ (and go to "Request New/Renew Accounts" or call 630-840-2345 - 24 hours a day).
* Make sure your Fermilab "training is up to date":http://www-esh.fnal.gov/pls/default/itp.html
* Make sure you have "LArTF Hazard Awareness training":http://www-esh.fnal.gov/pls/default/schedule.show_course_details?cid=8759.
** Once you have completed your trainings, send an email to "lartf_co@fnal.gov ":mailto:lartf_co@fnal.gov
* Read the *[[run_plan|Run Plan]]*
** Check the history to see recent changes: https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Run_plan/history
* Read the [[RC_-_Troubleshooting|Shifter Troubleshooting Page]] to find out what you should do in case of common problems during your shift

* Arrange for training with the "Run Coordinator":mailto:microboone_runco@fnal.gov
** This requires 1 hour of in-person training with a current or previous run coordinator. Group training sessions will be organised periodically that are accessible both locally at FNAL and remotely, although remote attendees will require access to the shared Slow Controls GUI and a separate Zoom screen-share. Contact the run coordinator to be put on the list for the next session. Other training sessions can be arranged in person with the run coordinators when available at FNAL; remote training is not available outside the scheduled sessions.
** After the initial training, a new shift is required to take 2 shadow shifts. Please have the shadowed shifters email the Run Coordinator confirm that the shadow shifts are complete.
** Ask the Run Coordinator or other expert to add your Kerberos principal to the MicroBooNE .k5login

h1. Shift Procedure

h2. Review Operations Wiki Updates page

* %{background:yellow; color:black}Changes in shift procedure and important notices for shifters are listed by date in the *[[OpsWikiUpdates|Operations Wiki Updates]]* page. Please review this list at the *beginning of every shift* and contact the relevant subsystem expert or Run Coordinator if you are not clear on new procedures.%

* Be sure to reread parts of the Operations Wiki that have been updated (or are new) since the last shift you took. The Operations Wiki Updates page should make it very simple to determine which material should be reviewed.

* If your last shift was before October 14th, 2015 (the date the Operations Wiki Update page was created), please read the entire Operations Wiki. Ideally this should be done before the beginning of your shift.

h2. Follow The Shift Turnover Procedures

* Please note that you are required to check in to your shift ten minutes before the official start of your shift, as listed on the elog. Otherwise, follow these procedures for communicating with the outgoing shifter: *[[shift_turnover|Shift Turnover Procedures]]*

h2. Log on to the control room computers

* On the mac, the user is ubooneshift, and the password is obtainable from the previous shifter or from the RunCo.

* On the Linux boxes, there are two cases *(You must be added to the .k5login file. "Contact us to help":mailto:microboone_runco@fnal.gov)*:
 
** Case 1. The screen has been locked. In this case enter your Kerberos password, followed by your Keberos principal (user name).
 
** Case 2. You have been logged out. In this case first enter the user as "ubooneshift", then enter your Kerberos password, and finally enter your Keberos principal (user name).

h2. REFRESH ALL OPEN BROWSERS and sign in to E-log, Google+

* Browsers open with instructions may have been changed! Make sure you have the latest instructions.

* SIGN INTO E-log WITH YOUR ACCOUNT: http://dbweb0.fnal.gov/ECL/uboone/E/index
All shift-related E-log entries should use the "Shift" category.

* Log on to the "ubooneshift google account":https://accounts.google.com/Login?hl=en with the password (obtainable from previous shifter or from RunCo), go to "Google hangouts":https://plus.google.com/, and then open up the "MicroBooNE Shift" conversation to communicate with "DAQ experts":mailto:kduffy@fnal.gov;adi@fnal.gov.

h2. Check for special run plans, else follow the routine shift procedure

* Open the *%{background:white; color:blue}[[run_plan|Run Plan]]%* page to see what the "Run Coordinator's":mailto:microboone_runco@fnal.gov plan for the day is.

* %{background:yellow; color:black}At the beginning of the shift call *MCR (x3721)* and let them know that the new MicroBooNE shifters are in ROC West.%

* At the beginning of shift, fill out the "Beginning of Shift":http://dbweb0.fnal.gov/ECL/uboone/E/create_entry?f=Beginning+of+Shift form on the E-log, and enter it under topic "Shift/Beginning of shift".

* During shift:
 
** Take runs according to the [[run_plan|Run Plan]], referencing [[RC - Guide]] for instructions about the run control. Contact DAQ on-call expert if there are any problems in data-taking (first following the recommendations in [[RC - Guide]]) - try Google Chat first and give the expert a phone call if they don't respond within a minute.
 
** Every four hours (*two* hours into the shift and *six* hours into the shift), fill out the "Checklist":http://dbweb0.fnal.gov/ECL/uboone/E/create_entry?f=Checklist form on the E-log, and enter it under topic "Shift/Checklist".
 
*** This checklist is used to ensure shifters are focusing on the right things during their shifts, to help track our noise issues as they develop over time, to catch things that "slip between the cracks" of our slow monitoring alarms, and to provide a summary of who to contact in case problems come up.
** Monitor the BNB status display (see [[Beam - Overview]] for instructions) and follow the directions in [[Beam - Guide]] about alarms or significant changes in beam conditions.
 
** Occasionally (every hour or so) check the "MCR E-log":https://www-bd.fnal.gov/Elog/?logNames=External+Beamlines&categoryNames=BNB for updates about the status of the BNB.
 
** Monitor the slow control display for recent alarms (see [[SLC - Guide]]). Check for alarming/tripping power supplies and call the relevant subsystem expert if you see major alarms. Acknowledge previously understood alarms, and new major alarms after contacting the relevant subsystem expert. In either case fill out the "Alarm Acknowledged":http://dbweb0.fnal.gov/ECL/uboone/E/create_entry?f=Alarm+Acknowledged form on the E-log. For minor alarms, send an email to the relevant subsystem expert instead.
 
*** For cryo-related alarms in the slow control display, before contacting anyone you should first verify that the cryo IFIX display is also showing an alarm. The cryo IFIX display is located on the opposite side of the NOvA shifter station (by the white boards along the side of the room). The machine with the IFIX display is the one closest to the door (but not the one facing the door that is used to control the displays on the walls). Use the default username; the password is obtainable from the previous shifter or from the RunCo. If the IFIX display is not showing an alarm as well, contact the Run Coordinator (phone call for major alarm, email for minor alarm) instead of MC-1 (as instructed in [[SLC - Guide]]).
 
** Look at the Online Monitor plots for the runs you take while on shift. See an example of things to monitor here: [[OM - Checklist]].
 
** Monitor data processing, a.k.a. PUBS. Make sure PUBS is running and that the computers on which PUBS is running are not overloaded. See the specific instructions at [[DM - Shifters]].
 
** Please check the quality of a data run using the [[Run Catalogue]] once a run finishes. Add in any comments about occurrences during the run.
** This replaces the *%{background:white; color:blue}"Run Information Document":https://docs.google.com/spreadsheets/d/1aVGAZrCYizhmduY4pqx5k29pVjyPRO7ydBJj95i0XfA/edit#gid=0%*
 
*** This document includes ASIC configuration settings as well as information about activity at LArTF during the collection of data. Please continue to update this document during early operations as it will be very helpful in helping us diagnose noise-related concerns, for instance. Please update this document during your shift and copy your updates into an E-log entry at the end of your shift.
 
*** Also note that noting operations-related items in the Run Information Document or making a note in the shifter Google chat window is not a replacement for an E-log entry. Any operations-related items of interest must be noted in the E-log, both *promptly* and with *specific information* (such as *which* alarms went into a bad state).
** %{background:yellow; color:black} Shifters need to notify Run Coordinator when experts are going to work in LArTF and also when the shifters are asked to help the experts in LArTF.%
 
*** Shifters must also fill out an E-log entry using the Start of Detector Access form, and fill out the End of Detector Access form when the work is complete.
 
*** We have a DEDICATED form for the weekly cryo walk-through. You do NOT have to inform the RC about these.
 
*** Be sure to follow the [[LArTF access procedures for shifts]].

* At the end of shift, fill out the "End of Shift":http://dbweb0.fnal.gov/ECL/uboone/E/create_entry?f=End+of+Shift form on the E-log, and enter it under topic "Shift/End of shift". Communicate with the incoming shifters (who should arrive at least 10 minutes before the end of the shift) the events of the shift, anything unusual they should be monitoring, and changes in the [[run_plan|Run Plan]] communicated to you by the Run Coordinator or DAQ experts.

h1. Shift Workstations

* [[Workstation Layout]]: this page shows what should be where, on the workstation monitors in front of you

h1. Run Control

* [[RC - Overview]]: How your run control screen should look like

* [[RC - Guide]]: How to start and stop a run
** (Old guide is [[ RC - OldGuide|here]])

* [[RC - ReconfigASIC]]: How to reconfigure ASICs.

* [[RC - CalibrationFanout]]: How to handle the calibration fanout

* [[RC - Troubleshooting]]: Troubleshooting for shift problems. What should I do if... ?

* [[DAQ_Expert_Documentation|DAQ Expert Documentation]]: Documentation for DAQ experts

* [[RC - Experts Only]]: Old DAQ Expert Documentation (out of date - use the new DAQ expert documentation from the link above!)

h1. Slow Controls/Monitoring

* [[SLC - Overview]]: An Introduction to the slow control screen

* [[SLC - Guide]]: What should I look at? How should it look like?

* [[SLC - Troubleshooting]]: What should I do if... ?

* [[SLC - Experts Only]]: Expert Documentation

h1. Online Monitoring Lizard

* [[OM - Checklist]] Checklist for each noise run

* [[OM - Errors]] A list of errors that can come up with the Big Red Box

* [[OM - What to look at]]: Shifters - Things to look at

* [[OM - Overview]]: Detailed guilde to the online monitoring screen

* [[OM - Guide]]: Access and technical information.

* [[OM - Troubleshooting]]: What should I do if... ?

* [[OM - Experts Only]]: Expert Documentation

h1. Run Catalogue

* Fill in the [[Run Catalogue]] as runs are taken, to check the quality of the data

h1. Data Management, Production, and PUBS

* [[DM - Shifters]] Shifters: read here first

* [[DM - Overview]] Overview of the DM system for non-experts

* [[DM - Troubleshooting]]: How to address DM problems

* [[DM - Experts Only]]: Guide to the Dark Side

* [[Production - Status]]: Summary of Production Activities and Plans

h1. Laser System

* [[Laser - Overview]]

* [[Laser - Guide]]

* [[Laser - Troubleshooting]]

* [[Laser - Experts Only]]:

h1. Light Collection/PMT System

* [[PMT - Overview]]

* [[PMT - Guide]]

* [[PMT - Troubleshooting]]

* [[PMT - Expert Info.]]

h1. Beam Monitoring

* [[Beam - Overview]]

* [[Beam - Guide]]

* [[Beam - Troubleshooting]]

* [[Beam - Experts Only]]

h1. Raspberry Pi Webcam

* [[Webcam - Overview]]

* [[Webcam - Guide]]

* [[Webcam - Troubleshooting]]

* [[Webcam - Experts Only]]

h1. Readout Electronics

* [[Readout Electronics - Overview]]

* [[Readout Electronics - Guide]]

* [[Readout Electronics - Troubleshooting]]

* [[Readout Electronics - Experts Only]]

h1. Time Projection Chamber (TPC) and Drift High Voltage (HV)

* [[TPC-Overview]]: This is a shifter guide to what are the relevant systems associated with the TPC

* [[TPC-Guide]]: Here is a step-by-step guide to monitoring things related to the operation of the TPC

* [[TPC-Troubleshooting]]: Quick guide to previously experienced problems with the TPC *(When in doubt call the TPC/HV expert)*

* [[TPC-Experts Only]]: TPC/HV Expert documentation can be found [[https://docs.google.com/document/d/16IeOlIUcblwuP2HzMcXSCTkygXg7JfI-NtQpw6fcSY4/edit?usp=sharing]] and should only ever be used by a trained TPC expert

* [[Cathode High Voltage - Overview]]: Overview of TPC cathode/drift high voltage

* [[Cathode High Voltage - Troubleshooting]]: Quick guide to previously experienced problems related to TPC drift high voltage *(When in doubt call the TPC/HV expert)*

* [[Cathode High Voltage - Experts Only]]: Deprecated. This information is contained in the google docs link above.

h1. Cryo

* [[Cryo-Procedures]]: What to do in the event of a cryo alarm.

h1. Purity Monitors

* [[Purity Monitors - Overview]]: A brief overview of the system

* [[Purity Monitors - Guide]]: A guide for shifters

* [[Purity Monitors - Experts Only]]: A guide for experts

h1. Nearline Monitoring

* [[Nearline Monitoring - Overview]]: A brief overview of the system

* [[Nearline Monitoring - Guide]]: A guide for shifters

* [[Nearline Monitoring - Experts Only]]: A guide for experts

h1. MiniBooNE

* [[MiniBooNE - Overview]]: A brief overview of the system

* [[MiniBooNE - Guide]]: A guide for shifters

h1. Online Event Display

* [[Online evd - Overview]]: A brief overview of the event display and what it is doing.

* [[Online evd - Guide]]: A guide to the functions of the display and how to use them

* [[Online evd - Troubleshooting]]: Solutions to common issues encountered

* [[Online evd - Experts Only]]: Guide for experts

h1. Argo Event Display

* [[Argo evd - Guide]]: A brief overview of the Argo event display and what it is doing.

* [[Argo evd - Troubleshooting]]: Solutions to common issues encountered

h1. Off-Site Shifts

* [[Off-Site Shifting Requirements]]: Requirements for off-site shifting

* [[Off-Site System Setup]]: A description of the requirements for off-site shift centers

* [[Off-Site Procedures]]: Instructions for off-site procedures

h1. Muon Counter System (MuCS)

* [[Muon Counter System - Experts Only]]

h1. Cosmic Ray Tagger System (CRT)

* [[Cosmic Ray Tagger - Overview]]

* [[Cosmic Ray Tagger - Guide]]

* [[Cosmic Ray Tagger - Experts Only]]

h1. Continuous Integration Validation

* [[CI Validation Instructions ]]

h1. Example System

* [[Example - Overview]]: This is basically what should the shifter be looking at (not necessarily touching), try to be explicit about what give what information about what to the shifter.

* [[Example - Guide]]: How should the shifter navigate through the different screens to retrieve the necessary information

* [[Example - Troubleshooting]]: What issues will the shifter experience and how do they fix them

* [[Example - Experts Only]]: This is where the "Expert Documentation" should live, but only for the time being, we will want to move this behind a password wall. For the time being please start to build the documentation here now but be mindful that it is public viewable.

h1. Call Lists



h2. "Expert call list":http://microboone-exp.fnal.gov/at_work/expert_call_list.html list":http://www-microboone.fnal.gov/at_work/expert_call_list.html (password-protected)



h2. "Collaboration contact list":http://www-microboone.fnal.gov/at_work/collaboration_contact_list.html (password-protected)

h1. Useful Links

* "MicroBooNE E-Log":http://dbweb0.fnal.gov/ECL/uboone/

* "Shift calender":http://dbweb0.fnal.gov/ECL/uboone/C/show_month : Look who is on shift or sign up for shifts

* "BNBStatus":http://www-bd.fnal.gov/synoptic/display/BNB/BNBStatus

* "MCR E-Log":https://www-bd.fnal.gov/Elog

* [[Computing-Support|Computing Support]] (shortcut links)

h1. Expert actions

* [[k5login]] Adding shifters to the .k5login

* [[Keytab]]: Automatically renew the kerbero tickets on evb and control room machines.

* [[Rack Power ON/OFF procedures]]: Follow this link to power on and off racks (mainly meant to guide during power outages)

* [[Procedures for Equipment Failure]]: Follow this link for procedures on how to deal with different type of equipment failures.

* [[MicroBooNE spares]]: Follow this link to see a list of MicroBooNE spares, location, and date last checked.

* [[HV trip recovery]]: Actions and checks following an HV trip to ensure the detector is healthy before ramping the HV again.

* [[Updating RunCat Instructions]]: Instructions for updating the list of run checks in RunCat.

* [[Direct access to Hoot Gibson]]: Log in as read-only user, execute SQL queries like a boss.