Project

General

Profile

What to do on shift » History » Version 45

« Previous - Version 45/64 (diff) - Next » - Current version
Diana Patricia Mendez, 08/28/2020 05:19 PM


What to do while on shift

Collaborators are required to check this page at least one week before the start of their shift and also on the day of the shift in case that something changed.

Shifters are strongly advised to familiarise with the monitoring screen layouts and with the shift procedures prior to their actual shift. Therefore, a shifter might want to do the following days ahead of the shift if

  • Monitoring from home: Contact the the current shifter via Slack or email to give this person a heads up about your intention to test the connections and get acquainted with what to do during the shift. If you think you might need help, ask the shifter if they can guide you through the steps and maybe even have a chat via the control room zoom. It might be possible that the shifter is unable to assist you at that moment, or that expert activity is scheduled. In that case, you might have to arrange another time for the test.
  • Monitoring from Fermilab's ROC-West (ROC-West is closed until further notice) Drop by ROC-West in advance to your shifts. Either the current shifter or other collaborator with previous experience will be happy to help you get up to speed. ROC-West is open 24/7 for everyone holding a Fermilab batch ID.
How do I know who is on shift?
  • Who is currently on shift? Check the ECL Who is on shift now section . You can also visit the #icarus-shift-operations channel in the SBN Slack to find out who the current shifter is by scrolling through the messages.
  • Who will or has been on shift? Check the ECL Shifts Calendar .


Shift changeover

From-home shift (from March 2020 until further notice)

The moment shift changeover happens mostly via 2 methods:
  • SBN Slack. Most of the shift changeover discussion happens in the #icarus-shift-operations channel. This channel has the benefit of having experts nearby.
  • Zoom room. Useful if verbal communication at handover is preferred or necessary. The connection details are at the beginning of DocDB 15944 (or in this alternate link)


Using ROCs (in person)

The ROC-West computers use what's called the three-point login where we used a shared computer account but authenticate against our individual Kerberos credentials. The shared username is sbn, which is the first step of logging in to the machines if they have just been started. Following this, you will be prompted for your Kerberos credentials. Type your Kerberos password, THEN your Kerberos principal (username). This will allow you to log into the machine, assuming you are registered as a shifter. If the machines are already on but locked, you just have to enter the two Kerberos portions of the three-point login.

During the shift change-over, the procedure will differ a bit depending on the location of the shifters.

Assuming you are both shifting from onsite:

  1. The previous shifter should lock the screens. Do this by clicking the power button at the very top of one of the monitors in each CR machine and clicking the lock symbol, as below.
    How to lock the screens for the FNAL control room
  2. The next shifter should then log in with their Kerberos credentials to begin taking the shift.

Assuming one shifter is remote:

  1. If the Fermilab shifter is leaving shift, they should lock the screens before leaving. If the Fermilab shifter is beginning shift, they should log in using their Kerberos credentials before the out-going shifter leaves.
  2. If the remote is leaving shift, they should close out of their windows and lock/log-out of the machines as per the protocol of their institution or defined by some collaboration specifications. (At the time of writing there are no remote control rooms, so we must be vague...). If the remote shifter is beginning shift, they should bring up the screens as defined by protocol.

If both shifters are remote:

  1. Both shifters follow step 2 of the above procedure.


Before the start of your shift

  1. Check that all screens are present and correct, and that there are no alarms.
  2. Talk to the outgoing shifter and find about any issues that occurred during the shift and anything else that you should be aware of during yours.
  3. Read the current running conditions in the Shift bulletin board and ECL entries for the last 24 hours, even if you were on shift on the previous day. Running conditions can change daily.
  4. Log in as yourself on any open ECL pages for speediness. This will save you time when filling the shift forms and reporting issues. You can of course also use your laptop to fill out ECL forms. Use your Fermilab Services password to log into the ECL.
  5. Call the main Fermilab control room (x3721) and let them know that the new ICARUS shifter has arrived and inform it of the control room you are shifting from. This is specially useful as people from the main control room might call you to inform about any work that might affect the beam status or the experiment's operations in general. Commented out as this is not required at the moment.

During the initial shifts: In fact at the moment we have brought up the EPICS Slow Controls monitoring on CR01 (the second set of 4 screens) as well as the main instance on CR02. Bringing up the DCS on CR01 follows the same procedure as for CR02. If there is an issue with either computer, shifters should make note of this in the ECL and use the other instance instead, following the procedure here: https://cdcvs.fnal.gov/redmine/projects/icarus-operations/wiki/Help!_I'm_on_shift_and_I'm_trying_to#-load-the-VNC-sessions . An expert should be called if none of the computers are working. Commented out as there are no in-person shifts.


Throughout the shift

Below is a list of the ECL forms that you should fill out throughout your shift. It is your responsibility to fill out all these forms during every shift. There are also other forms that you should fill out in certain situations and others that will be filled by experts. To fill up a form:

  1. Log into the ICARUS ECL.
  2. Click on New entry, in the left vertical menu.
  3. A new window will pop up.
  4. Select a form from the Form menu.
  5. Fill up the fields. Some comments to help you complete the form are shown on the left of each field.

In addition, please make sure to make an ECL post regarding any activity or issue you encounter while on shift. It is always preferred to have a lot of information instead of none. If you contact an expert please fill the Expert Contact form in ECL. Likewise, if you are an expert who has been contacted and/or are working on solving an issue, please fill the Expert Activity form.


Main ECL forms to be filled out during every shift

When Form Category Tag Instructions
Before the previous shifter leaves Shift check-in General Control Room, General
Immediately after shift check-in Start of shift General Control Room
Right after the start of day shift Daily ICARUS Access General Access, Work Plan Day shift only
At the beginning and at the middle of the shift Control room checklist General Control Room
Once every hour PMT HV Monitoring General PMT, Monitoring PMT HV Monitoring Wiki
Once every hour IFIX Cryogenics (Cryogenic Stability Checklist) General Control Room, Cryogenics Pressures can be found in top right of iFix screen, see Fig. 1 in Cryogenics_checklist
Every other hour DAQ Commissioning Checklist General Run Info, DAQ
In the last hour of your shift Shift summary General Control Room
After the new shifter has arrived and you have handed things over Shift check-out General Control Room, General

During the Cooldown/Commissioning: Please see the Shift Bulletin Board, specifically the entry labeled Taking runs during the cool down in bright bold red. Make ECL entries every 2 hours on the status of the run ("DAQ Commisioning Form") and another ECL entry each for if the run is brought down (explaining why) and one when a new run is started (Run_Start form). You may be asked to alter the schedule at which you post these forms and/or told you can hold off on posting this form during DAQ work/testing, but if not asked otherwise you should go with this. Thanks!


Suggested schedule to fill out main ECL shift forms

Time into shift Forms to complete
-0:15 - 0:00 Shift check-in
0:00 - 0:15 Start of shift
0:15 - 0:30 Control room checklist
0:30, 1:30, ..., 7:30 IFIX cryogenics (Cryogenic Stability Checklist)
0:30, 1:30, ..., 7:30 PMT HV Monitoring Form
1:00, 3:00, ..., 7:00 DAQ Commissioning Checklist
4:15 - 4:30 Control room checklist
7:00 - 8:00 Shift summary
8:00 - 8:15 Shift check-out


Recorded tutorials

  • Cryogenics An iFix tutorial for shifters can be found in the SBN DocDB 19190. You'll have to access like icarus. This video goes through the steps to connect to the iFix machine and it shows what to monitor during comissioning.


Good shifter practice

  • Show up to your shift 15 minutes before the scheduled start. You are also expected to be able to stay 15 minutes beyond the scheduled end of your shift.
  • Remain in the control room at all times except for justified brief absences of less than 15 minutes or in case of emergency (fire alarms, tornado warnings, etc).
  • Add an ECL entry for all the activities and issues that arise during your shift.
  • Add clear explanations to the ECL entries and add screen shoots if possible.
  • Add the relevant tags to the ECL entries (do not forget to click both the Add and Update buttons). This makes it easier for the experts and run coordinators to identify subjects.