What to do on shift » History » Version 62
What to do while on shift¶
Collaborators are required to check this page at least one week before the start of their shift and also on the day of the shift in case that something changed.
Shifters are strongly advised to familiarise with the monitoring screen layouts and with the shift procedures prior to their actual shift. Therefore, a shifter might want to do the following days ahead of the shift if
- Monitoring from home: Contact the the current shifter via Slack or email to give this person a heads up about your intention to test the connections and get acquainted with what to do during the shift. If you think you might need help, ask the shifter if they can guide you through the steps and maybe even have a chat via the control room zoom. It might be possible that the shifter is unable to assist you at that moment, or that expert activity is scheduled. In that case, you might have to arrange another time for the test.
- Monitoring from Fermilab's ROC-West (ROC-West is closed until further notice) Drop by ROC-West in advance to your shifts. Either the current shifter or other collaborator with previous experience will be happy to help you get up to speed. ROC-West is open 24/7 for everyone holding a Fermilab batch ID.
- Who is currently on shift? Check the ECL Who is on shift now section . You can also visit the #icarus-shift-operations channel in the SBN Slack to find out who the current shifter is by scrolling through the messages.
- Who will or has been on shift? Check the ECL Shifts Calendar .
- Table of contents
- What to do while on shift
- Shift changeover
- Phone calls on shift
- Before the start of your shift
- Throughout the shift
- Good shifter practice
From-home shift (from March 2020 until further notice)¶The moment shift changeover happens mostly via 2 methods:
- SBN Slack. Most of the shift changeover discussion happens in the #icarus-shift-operations channel. This channel has the benefit of having experts nearby.
- Zoom room. Useful if verbal communication at handover is preferred or necessary. The connection details are at the beginning of DocDB 15944 (or in this alternate link)
For the Incoming Shifter
Shortly after taking over the shift, the Event Display should be restarted. Close the Event Display that's open and relaunch it from the launchEVD icon located on the screen (at time of writing it's an icon on the left side of the screen and part way down... you may have to scroll down on the screen to see it). See the Event Display Wiki for more info.
For the Outgoing ShifterWhen closing your sessions to sign out:
- Please press the DISCONNECT button in the top right corner of the VNC sessions before closing the Web browser (e.g. Firefox) tabs. It is in the NoVNC menu on the left side of the screen (it may be just an arrow that you have to click to expand). The disconnect button is on the bottom, see the NoVNC Wiki for slightly more info as needed!
- For the Cryogenics Remote Desktop (iFix) connection ONLY, please sign off of your logged in session before closing your Remote Desktop connection.
Using ROCs (in person)¶
The ROC-West computers use what's called the three-point login where we used a shared computer account but authenticate against our individual Kerberos credentials. The shared username is sbn, which is the first step of logging in to the machines if they have just been started. Following this, you will be prompted for your Kerberos credentials. Type your Kerberos password, THEN your Kerberos principal (username). This will allow you to log into the machine, assuming you are registered as a shifter. If the machines are already on but locked, you just have to enter the two Kerberos portions of the three-point login.
During the shift change-over, the procedure will differ a bit depending on the location of the shifters.
Assuming you are both shifting from onsite:
- The previous shifter should lock the screens. Do this by clicking the power button at the very top of one of the monitors in each CR machine and clicking the lock symbol, as below.
- The next shifter should then log in with their Kerberos credentials to begin taking the shift.
Assuming one shifter is remote:
- If the Fermilab shifter is leaving shift, they should lock the screens before leaving. If the Fermilab shifter is beginning shift, they should log in using their Kerberos credentials before the out-going shifter leaves.
- If the remote is leaving shift, they should close out of their windows and lock/log-out of the machines as per the protocol of their institution or defined by some collaboration specifications. (At the time of writing there are no remote control rooms, so we must be vague...). If the remote shifter is beginning shift, they should bring up the screens as defined by protocol.
If both shifters are remote:
- Both shifters follow step 2 of the above procedure.
Phone calls on shift¶
In order to make phone calls during the shift, the shifters are required to log in to the Google Voice Shifter account (https://voice.google.com/about).
Password: See ECL
Shifter Telephone: +1(631) 894-4173
Google Voice account works on US based calls only and it can be used both for mobile and landline numbers. Phone numbers can be dialed via the keypad provided by the application.
Shifters in the US can use both mobile or browser app to make calls, while Europe based shifters can only do that via browser application.
The shifters are required to be logged in throughout his/her shift in order to be able to pick up calls. Multiple Google accounts can be used at the same time in different tabs, hence shifters can use their personal accounts for other Google products (Gmail, YouTube, etc...).
If the Shifter receives a phone call and he/she misses it, the call will be first forwarded to the Run Coordinator, then to deputy Run Coordinator and lastly to the Operation Support Coordinator.
Shifters are required to log off from the account once the following shifter checked in in order to prevent possible calls mismanagement (e.g. picking up/rejecting a call for the shifter on duty).
Before the start of your shift¶
- Check that all screens are present and correct, and that there are no alarms.
- Talk to the outgoing shifter and find about any issues that occurred during the shift and anything else that you should be aware of during yours.
- Read the current running conditions in the Shift bulletin board and ECL entries for the last 24 hours, even if you were on shift on the previous day. Running conditions can change daily.
- Log in as yourself on any open ECL pages for speediness. This will save you time when filling the shift forms and reporting issues. You can of course also use your laptop to fill out ECL forms. Use your Fermilab Services password to log into the ECL.
- Log into the Shifter Google Voice account, call the main Fermilab control room (+1 630 840 3721) and let them know that the new ICARUS shifter has arrived.
During the initial shifts: In fact at the moment we have brought up the EPICS Slow Controls monitoring on CR01 (the second set of 4 screens) as well as the main instance on CR02. Bringing up the DCS on CR01 follows the same procedure as for CR02. If there is an issue with either computer, shifters should make note of this in the ECL and use the other instance instead, following the procedure here: https://cdcvs.fnal.gov/redmine/projects/icarus-operations/wiki/Help!_I'm_on_shift_and_I'm_trying_to#-load-the-VNC-sessions . An expert should be called if none of the computers are working. Commented out as there are no in-person shifts.
Throughout the shift¶
Below is a list of the ECL forms that you should fill out throughout your shift. It is your responsibility to fill out all these forms during every shift. There are also other forms that you should fill out in certain situations and others that will be filled by experts. To fill up a form:
- Log into the ICARUS ECL.
- Click on New entry, in the left vertical menu.
- A new window will pop up.
- Select a form from the Form menu.
- Fill up the fields. Some comments to help you complete the form are shown on the right of each field. You may need to make the form pop-up window larger to see these instructions/comments.
In addition, please make sure to make an ECL post regarding any activity or issue you encounter while on shift. It is always preferred to have a lot of information instead of none. If you contact an expert please fill the Expert Contact form in ECL. Likewise, if you are an expert who has been contacted and/or are working on solving an issue, please fill the Expert Activity form.
Main ECL forms to be filled out during every shift¶
|Before the previous shifter leaves||Shift check-in||General||Control Room, General|
|Immediately after shift check-in||Start of shift||General||Control Room|
|Right after the start of day shift||Daily ICARUS Access||General||Access, Work Plan||Day shift only|
|Once every hour||PMT HV Monitoring||General||PMT, Monitoring||PMT HV Monitoring Wiki|
|Once every hour||IFIX Cryogenics (Cryogenic Stability Checklist)||General||Control Room, Cryogenics||Pressures can be found in top right of iFix screen, see Fig. 1 in Cryogenics_checklist|
|Once every hour||Cathode HV Checklist||General||Drift HV, Monitoring||Read the instructions on the right of each field on form! Also see Cathode HV Wiki|
|Every other hour||DAQ Commissioning Checklist||General||Run Info, DAQ|
|Every other hour||TPC Power Supply Checklist**||General||DCS, Monitoring, TPC||See "TPC Power Supply checks every two hours" on the Shift Bulletin board|
|After the new shifter has arrived and you have handed things over||Shift check-out||General||Control Room, General|
**: Use the general form with the listed tags. Run the check script and put the results in the ECL. Take actions as defined in the Shift Bulletin Board entry, if needed.
During the Cooldown/Commissioning: Make ECL entries every 2 hours on the status of the run ("DAQ Commisioning Form") and another ECL entry each for if the run is brought down (explaining why) and one when a new run is started (Run_Start form). You may be asked to alter the schedule at which you post these forms and/or told you can hold off on posting this form during DAQ work/testing, but if not asked otherwise you should go with this. Thanks!
Suggested schedule to fill out main ECL shift forms¶
|Time into shift||Forms to complete|
|-0:15 - 0:00||Shift check-in|
|0:00 - 0:15||Start of shift|
|0:15 - 0:30||Daily ICARUS Access (day shift only)|
|0:30, 1:30, ..., 7:30||IFIX cryogenics (Cryogenic Stability Checklist)|
|0:30, 1:30, ..., 7:30||PMT HV Monitoring Form|
|0:30, 1:30, ..., 7:30||Cathode HV Checklist|
|1:00, 3:00, ..., 7:00||DAQ Commissioning Checklist|
|1:00, 3:00, ..., 7:00||TPC Power Supply Check*|
|8:00 - 8:15||Shift check-out|
*: No official form, but run the script and put the results in the ECL using the tags listed above. Take actions as defined in the Shift Bulletin Board entry, if needed.
To see the tutorials, you should have accessed the SBN document database (DocDB) as icarus.
- Cryogenics An iFix tutorial for shifters can be found in the SBN DocDB 19190. This video goes through the steps to connect to the iFix machine and it shows what to monitor during commissioning.
- PMT A PMT tutorial for shifters can be found in the SBN DocDB 19441. This video goes through the steps to connect to the icarus evb 06, which currently runs the PMT EPICS. A video showing what to monitor during commissioning will also be uploaded soon.
Good shifter practice¶
- Show up to your shift 15 minutes before the scheduled start. You are also expected to be able to stay 15 minutes beyond the scheduled end of your shift.
- Remain in the control room at all times except for justified brief absences of less than 15 minutes or in case of emergency (fire alarms, tornado warnings, etc).
- Add an ECL entry for all the activities and issues that arise during your shift.
- Add clear explanations to the ECL entries and add screen shoots if possible.
- Add the relevant tags to the ECL entries (do not forget to click both the Add and Update buttons). This makes it easier for the experts and run coordinators to identify subjects.