What to do on shift » History » Version 77

« Previous - Version 77/110 (diff) - Next » - Current version
Andrea Scarpelli, 03/02/2021 10:38 AM

What to do while on shift

Collaborators are required to check this page at least one week before the start of their shift and also on the day of the shift in case that something changed.

Shifters are strongly advised to familiarise with the monitoring screen layouts and with the shift procedures prior to their actual shift. Therefore, a shifter might want to do the following days ahead of the shift if

  • Monitoring from home: Contact the the current shifter via Slack or email to give this person a heads up about your intention to test the connections and get acquainted with what to do during the shift. If you think you might need help, ask the shifter if they can guide you through the steps and maybe even have a chat via the control room zoom. It might be possible that the shifter is unable to assist you at that moment, or that expert activity is scheduled. In that case, you might have to arrange another time for the test.
  • Monitoring from Fermilab's ROC-West (ROC-West is closed until further notice) Drop by ROC-West in advance to your shifts. Either the current shifter or other collaborator with previous experience will be happy to help you get up to speed. ROC-West is open 24/7 for everyone holding a Fermilab batch ID.
How do I know who is on shift?
  • Who is currently on shift? Check the ECL Who is on shift now section . You can also visit the #icarus-shift-operations channel in the SBN Slack to find out who the current shifter is by scrolling through the messages.
  • Who will or has been on shift? Check the ECL Shifts Calendar .

Shift changeover

From-home shift (from March 2020 until further notice)

The moment shift changeover happens mostly via 2 methods:
  • SBN Slack. Most of the shift changeover discussion happens in the #icarus-shift-operations channel. This channel has the benefit of having experts nearby.
  • Zoom room. Useful if verbal communication at handover is preferred or necessary. The connection details are at the beginning of DocDB 15944 (or in this alternate link)

For the Incoming Shifter

Shortly after taking over the shift, the Event Display should be restarted. Close the Event Display that's open and relaunch it from the launchEVD icon located on the screen (at time of writing it's an icon on the left side of the screen and part way down... you may have to scroll down on the screen to see it). See the Event Display Wiki for more info.

For the Outgoing Shifter

When closing your sessions to sign out:
  • Please press the DISCONNECT button in the top right corner of the VNC sessions before closing the Web browser (e.g. Firefox) tabs. It is in the NoVNC menu on the left side of the screen (it may be just an arrow that you have to click to expand). The disconnect button is on the bottom, see the NoVNC Wiki for slightly more info as needed!

  • For the Cryogenics Remote Desktop (iFix) connection ONLY, please sign off of your logged in session before closing your Remote Desktop connection.

Using ROCs (in person)

The ROC-West computers use what's called the three-point login where we used a shared computer account but authenticate against our individual Kerberos credentials. The shared username is sbn, which is the first step of logging in to the machines if they have just been started. Following this, you will be prompted for your Kerberos credentials. Type your Kerberos password, THEN your Kerberos principal (username). This will allow you to log into the machine, assuming you are registered as a shifter. If the machines are already on but locked, you just have to enter the two Kerberos portions of the three-point login.

During the shift change-over, the procedure will differ a bit depending on the location of the shifters.

Assuming you are both shifting from onsite:

  1. The previous shifter should lock the screens. Do this by clicking the power button at the very top of one of the monitors in each CR machine and clicking the lock symbol, as below.
    How to lock the screens for the FNAL control room
  2. The next shifter should then log in with their Kerberos credentials to begin taking the shift.

Assuming one shifter is remote:

  1. If the Fermilab shifter is leaving shift, they should lock the screens before leaving. If the Fermilab shifter is beginning shift, they should log in using their Kerberos credentials before the out-going shifter leaves.
  2. If the remote is leaving shift, they should close out of their windows and lock/log-out of the machines as per the protocol of their institution or defined by some collaboration specifications. (At the time of writing there are no remote control rooms, so we must be vague...). If the remote shifter is beginning shift, they should bring up the screens as defined by protocol.

If both shifters are remote:

  1. Both shifters follow step 2 of the above procedure.

Phone calls on shift

In order to make phone calls during the shift, the shifters are required to log in to the Google Voice Shifter account (
Account name:
Password: See ECL
Shifter Telephone: +1(631) 894-4173‬

Google Voice account works on US based calls only and it can be used both for mobile and landline numbers. Phone numbers can be dialed via the keypad provided by the application.
Shifters in the US can use both mobile or browser app to make calls, while Europe based shifters can only do that via browser application.
The shifters are required to be logged in throughout his/her shift in order to be able to pick up calls. Multiple Google accounts can be used at the same time in different tabs, hence shifters can use their personal accounts for other Google products (Gmail, YouTube, etc...).
If the Shifter receives a phone call and he/she misses it, the call will be first forwarded to the Run Coordinator, then to deputy Run Coordinator and lastly to the Operation Support Coordinator.
Shifters are required to log off from the account once the following shifter checked in in order to prevent possible calls mismanagement (e.g. picking up/rejecting a call for the shifter on duty).
If the shifters need to call a US based expert (see +1 prefix before their number in the Expert Contact List), please use the Google Voice number; if the experts miss the call, once available they will know that the shifter required help.
If the experts are not US based, the current solution is to call them using Whatsapp/Telegram or other applications that require WiFi/4G/5G in order to minimize the costs.
IMPORTANT For security reasons a 2 factors authorization is sometimes required to log in into the Google Voice account. If you are requested the pin, notify the current shifter of this and he will communicate it to you. Pins are received as a text message or as a notification on the shifter's Google voice phone number. Please ask @icarus_shift_help in case of problems.

Before the start of your shift

  1. Check that all screens are present and correct, and that there are no alarms.
  2. Talk to the outgoing shifter and find about any issues that occurred during the shift and anything else that you should be aware of during yours.
  3. Read the current running conditions in the Shift bulletin board and ECL entries for the last 24 hours, even if you were on shift on the previous day. Running conditions can change daily.
  4. Log in as yourself on any open ECL pages for speediness. This will save you time when filling the shift forms and reporting issues. You can of course also use your laptop to fill out ECL forms. Use your Fermilab Services password to log into the ECL.
  5. Log into the Shifter Google Voice account, call the main Fermilab control room (+1 630 840 3721) and let them know that the new ICARUS shifter has arrived.

Throughout the shift

Below is a list of the ECL forms that you should fill out throughout your shift. It is your responsibility to fill out all these forms during every shift. There are also other forms that you should fill out in certain situations and others that will be filled by experts. To fill up a form:

  1. Log into the ICARUS ECL.
  2. Click on New entry, in the left vertical menu.
  3. A new window will pop up.
  4. Select a form from the Form menu.
  5. Fill up the fields. Some comments to help you complete the form are shown on the right of each field. You may need to make the form pop-up window larger to see these instructions/comments.

In addition, please make sure to make an ECL post regarding any activity or issue you encounter while on shift. It is always preferred to have a lot of information instead of none. If you contact an expert please fill the Expert Contact form in ECL. Likewise, if you are an expert who has been contacted and/or are working on solving an issue, please fill the Expert Activity form.

Main ECL forms to be filled out during every shift

When Form Category Tag Instructions
Before the previous shifter leaves Shift check-in General Control Room, General
Immediately after shift check-in Start of shift General Control Room
Right after the start of day shift Daily ICARUS Access General Access, Work Plan Day shift only (weekdays)
Once every hour PMT HV Monitoring General PMT, Monitoring PMT HV Monitoring Wiki
Once every hour IFIX Cryogenics (Cryogenic Stability Checklist) General Control Room, Cryogenics Pressures can be found in top right of iFix screen, see Fig. 1 in Cryogenics_checklist
Once every hour Cathode HV Checklist General Drift HV, Monitoring Read the instructions on the right of each field on form! Also see Cathode HV Wiki
Every other hour DAQ Commissioning Checklist General Run Info, DAQ
Every other hour TPC Commissioning monitoring checklist** General DCS, Monitoring, TPC See "TPC Power Supply checks every two hours" on the Shift Bulletin board
Every other hour BNB simplified checklist General Monitoring, Beam More info on Beam display page
After the new shifter has arrived and you have handed things over Shift check-out General Control Room, General

**: Use the general form with the listed tags. Run the check script and put the results in the ECL. Take actions as defined in the Shift Bulletin Board entry, if needed.

During the Cooldown/Commissioning: Make ECL entries every 2 hours on the status of the run ("DAQ Commisioning Form") and another ECL entry each for if the run is brought down (explaining why) and one when a new run is started (Run_Start form). You may be asked to alter the schedule at which you post these forms and/or told you can hold off on posting this form during DAQ work/testing, but if not asked otherwise you should go with this. Thanks!

The shifter has among his duties to take purity runs. This is in general done by the swing shifter around 6 pm CST. In general, the shifter is informed of when to take purity runs. The instruction can be found on the dedicated wiki. For the swing shifter, it is a good practice to confirm with the commissioner or deputy commissioner when purity runs are due.

Suggested schedule to fill out main ECL shift forms

Time into shift Forms to complete
-0:15 - 0:00 Shift check-in
0:00 - 0:15 Start of shift
0:15 - 0:30 Daily ICARUS Access (day shift only)
0:30, 1:30, ..., 7:30 IFIX cryogenics (Cryogenic Stability Checklist)
0:30, 1:30, ..., 7:30 PMT HV Monitoring Form
0:30, 1:30, ..., 7:30 Cathode HV Checklist
1:00, 3:00, ..., 7:00 DAQ Commissioning Checklist
1:00, 3:00, ..., 7:00 TPC Power Supply Check*
0:00, 2:00, ..., 6:00 BNB simplified checklist
8:00 - 8:15 Shift check-out

*: No official form, but run the script and put the results in the ECL using the tags listed above. Take actions as defined in the Shift Bulletin Board entry, if needed.

Recorded tutorials

To see the tutorials, you should have accessed the SBN document database (DocDB) as icarus.

  • Cryogenics An iFix tutorial for shifters can be found in the SBN DocDB 19190. This video goes through the steps to connect to the iFix machine and it shows what to monitor during commissioning.
  • PMT A PMT tutorial for shifters can be found in the SBN DocDB 19441. This video goes through the steps to connect to the icarus evb 06, which currently runs the PMT EPICS. A video showing what to monitor during commissioning will also be uploaded soon.

Good shifter practice

  • Show up to your shift 15 minutes before the scheduled start. You are also expected to be able to stay 15 minutes beyond the scheduled end of your shift.
  • Remain in the control room at all times except for justified brief absences of less than 15 minutes or in case of emergency (fire alarms, tornado warnings, etc).
  • Add an ECL entry for all the activities and issues that arise during your shift.
  • Add clear explanations to the ECL entries and add screen shoots if possible.
  • Add the relevant tags to the ECL entries (do not forget to click both the Add and Update buttons). This makes it easier for the experts and run coordinators to identify subjects.