Shift bulletin board » History » Version 277

« Previous - Version 277/504 (diff) - Next » - Current version
Andrea Scarpelli, 03/01/2021 11:22 PM

Shift bulletin board

Don't forget to read both the TEMPORARY and PERMANENT conditions

Colour guide for this page

Current Condition Temporary Conditions New and Permanent Conditions Attention

General comments from the Run Coordinators

Role Name Until (Date) Email Phone Number
Commissioning coordinator Angela Fava January 1st, 2021 8am afava at (xxx) xxx-xxx
Deputy commissioning Coordinator Andrea Scarpelli January 1st, 2021 8am ascarpell at (xxx) xxx-xxx
Run coordinator Name Surname January 1st, 2022 8am someone at something (xxx) xxx-xxx

ICARUS DAQ Expert (also there is the tag @sbn_daq_experts on your post on Slack)

Expert for the week of: 8am FNAL time on February 8th to 8am FNAL time on March 1st

Antoni Aduszkiewicz (@Antoni Aduszkiewicz on Slack)

You may also contact all experts on Slack via @sbn_daq_experts.

ICARUS Shift Help Team (tag @icarus-shift-help on your post on Slack)

Experts names: Francesco Poppi / Bruce Howard / Wenqiang Gu / Wooyoung Jang

Also see ECL calendar/shift list

ICARUS PMT Expert (tag @icarus_pmt_expert on your post on Slack)

Experts for the shift block: from February 22th to March 1st

Also see ECL calendar/shift list

00:00 - 11:59 CST / 12:00 - 23:59 CST : Niccolò Moggi / Andrea Scarpelli


Shifters should resume calling the Fermilab main control room when they fill out their start shift form. The number to call is in the Start Shift form.


Then please write a general ECL entry logging the details of the call, and send this ECL entry as an email to Angela Fava, Andrea Scarpelli, and Donatella Torretta.

*Google Voice Shifter account (Updated 6 January 2021)
Shifters now have access to a Google Voice account in order to make/receive calls.
Please see for Account name and ECL for password.
Remember to log in before your shift and to log out once the following shifter has checked in.
If the shifters need to call US based experts, please use this account in order to let the experts know who is calling.
If the experts are not US based, it is suggested to use Whatsapp/Telegram or other applications that require WiFi/4G/5G in order to minimize the costs.

No data taken tonight(updated 1st March 2021)

The daq checklist is not due, as we are not going to take data for the night of March 1st until the morning of March 2nd. The shifter will be contacted by the experts with new information on how to proceed. All other subsystems are supposed to be monitored as usual.

Run EventBuilder PDU check script from the terminal in Slow Controls machine (updated 15 December 2020)

Instructions starting 15 December 2020 for the time being.

From a terminal in the Slow Controls VNC, run the command

and report its result in the ECL. A suggested title would be "Event Builder PDU Check"

Specifically, you want to check the results of the lines EVB rack, PDU Bank 1: and EVB rack, PDU Bank 2:

This should be run every 2 hours.

If the value reported in the line for Bank 1 is > 20 A or the value reported in the line for Bank 2 > 2 A, then call and alert the Commissioning Coordinator (Angela Fava) or Deputy (Andrea Scarpelli), whose numbers can be found on the Expert Call List.

New BNB status checklist (Updated to 7th Feb. 2021)

Shifters should now fill a checklist to monitor the BNB status. The checklist name is "BNB simplified checklist". It is due every two hours.

BNB status(Updated to 26th Feb. 2021)

BNB status

Restart Event Display at the start of EVERY shift (added 12 November 2020, updated last 10 December 2020)

Reminder to shifters to close any open Event Display and re-open it using the icon on the desktop. This should be done at the start of EVERY shift until further notice. The wiki on the event display is given here and some suggestions for settings are in this ECL entry, see the comment of this ECL post as well. This is a useful tool for looking at events in the cryostats!

NOTE on 10 DECEMBER 2020: Really we want it to be restarting more frequently than every 8 hours. Wes has set up the script to automatically restart the event display every 2 hours instead. We apologize for this inconvenience, but for now this really needs to restart to clear out it's memory. It's important.

OnMon updates in progress (added 10 December 2020)

A test version of reference plots has been added to the online monitoring plots. PLEASE NOTE: there seems to be some mirroring in the reference plots at the moment.

If Grafana is showing data disk(s) >=95% full (added 13 November 2020)

If Grafana shows any of the data disks at or above 95% full (visible near the lower right of the Shifter Grafana page), then please call Wes K, who can be found on the Experts On Call document.

TPC Power Supply checks every two hours (updated 26 December 2020, expert on-call updated as needed)

We're requesting for now that a TPC Power Supply check be done every 2 hours. Basically you run the script


from the terminal in the Slow Controls VNC (or if the terminal that had been running it isn't there for some reason you can open a new one). Then take a screenshot and post in the ECL as a general post with the DCS, Monitoring, and TPC tags]

Best case is it says there are 0 complaints. If there are tiny temperature complaints these are usually okay. If there are temperature warnings > 5 degrees from reference value or > 50 degrees, or for all other warnings please write a message in the shift Slack channel and tag


If there is no response to Slack after about 15 minutes, then call an expert. If any temperature is >51 degrees or there are warnings on the 12V line (i.e. icarus_tpcps_*_p12), then call an expert.

The primary on-call expert for this week is Guang Meng, whose number is in the Experts On Call document and you can also get clicking on "Who is on shift now" in the ECL page side menu. In the case where someone has trouble reaching the primary on-call expert (i.e. they are not answering or responding), then call down the list of TPC power supply experts to attempt to reach an expert.

IMPORTANT!! Ongoing issues with East cryostat pressures & note about who to call (updated 4 January 2021)

Issues with a re-condenser in the EAST module means the pressure may be higher than usual. This is known to the experts.

For the East only, the higher limit to call Claudio is if it goes above 150 mbar relative. For the WEST module, take 140 mbar relative as an upper limit beyond which to call Claudio.

Call Claudio before the Cryo Coordinator if the module's pressure goes out of range. For all other alarms the rule does not change about which expert to call.

Live Event Display (added 21 October 2020)

See the Live Event Display Wiki Page for instructions. Note in particular the "Notes For Usage" passage explaining some important details about using the event display. Also see this ECL entry for some suggested configuration information.

Shifters are encouraged to put the summary of their shift and info passed to next shifter in their shift check-out form! (added 21 October 2020)

The shift check-out form has a box for general comments. Generally, shifters are encouraged to enter the summary they gave to incoming shifter and any notes there as well, and not just in Slack.

CURRENT DAQ CONFIGURATION (updated 14 January 2021)

New conditions for the run with respect to previously:
  • Uses the icons from the Desktop folder on the DAQ VNC (if it's closed for some reason, open the program Caja under the ICARUS program menu and then make sure you are already in or go to the Desktop area in the file browser that opens)
  • WE ARE NOW USING THE CONFIGURATION DATABASE! Make sure the "Use database configurations" is checked when you select a configuration.
  • Select the standardTPCWest configuration for standard shifter runs. You can type "standard" in the filter box to help find the right one. There will be a version number after the config name: usually, there should only be one valid configuration with that name that is visible, but if there are multiple ones use the latest-numbered one. Please, note that we use the database configuration also for the purity runs. In this case, the configurations to select are lifetimeTPCEast and lifetimeTPCWest. Also in this case if there are multiple numbered ones, the latest numbered-version is the one to use.
  • Use the boot_split.txt boot file
  • Use ONLY WE and WW TPC components (icarustpcwe* and icaruxtpcww*), all listed CRT components, and all PMTs. NO other components. The new configuration DB should automatically select all components and only contain the components we want to run, but please double-check!

Shifters should go back to filling out the DAQ Checklist and monitoring Grafana and OnMon.

There is a CRT data rate plot on the shifter page in Grafana. If you notice sudden changes in this rate, please make a General post on the ECL and attach a screenshot and words with what you noticed! Send the ECL form as an email to Antoni Aduszkewicz, Tyler Boone, and Biswaranjan Behera. This is possible by using the "Email new entry to" box. Type part or all of a name and click their name and "add ->" to add each person to the form.

If the run crashes, shifters should go through the usual steps of trying a restart first, but if runs are also crashing after attempting 3 restarts and/or you are having trouble after struggling with the DAQ for 15-20 minutes, then communicate with a DAQ expert (see the current on-call and Slack information at the top of the Bulletin Board).

In case of multiple crashes or needing to restart the run, the shifter can try to run with just the following 4 CRT components (icaruscrt01nni, icaruscrt01nno, icaruscrt03wci, and icaruscrt03wco).

IMPORTANT!! Changes to iFix instructions for shifters (added 28 August 2020, updated slightly 2 November 2020)

To check for alarms in the system, shifters should look at the Alarm Summary in the top right of the iFix window as well. This shows two types of alarms, regular and critical. For regular alarms the bottom box turns red. For CRITICAL alarms, the top box turns red. See picture below. On alerting cryo experts:

CRITICAL Alarms: call cryo expert whenever this happens (any hour of day or day of week)
Regular Alarms: call cryo expert between the hours 08:00 and 17:00 Fermilab time. If your shift is overnight, pass word on to the next shifters to call when it is again between these hours, so long as the alarm doesn't become CRITICAL.

If pressures are outside the OKAY ranges, call a cryo expert any time of day or day of the week! (NOTE: the instructions were recently updated to call Claudio first in these situations)

Any time you see any alarm make a separate post in the ECL about the alarm noticed and any actions taken.

iFix Alarm Summary

Current Running Conditions

Commissioning Run Schedule (Added 19 August, 2020)

To better plan around activities related to commissioning tests and to help shifters be aware of what activities might be happening, there is now a spreadsheet

Temporary Conditions

If a run is going and suddenly the GUI becomes disconnected! (Added 29 July 2020)

We are trying to keep track of a recent issue we've noticed where a run is going and suddenly the Run Control GUI looks blank and as if it wasn't started (the only button to click on is "Start DAQInterface"). In these scenarios, the run is probably still going and just the GUI has lost connection to the run. We are trying to understand these more.

At the moment, recovering from these are an expert activity. If this happens, please get in touch with Bruce Howard or Wes Ketchum (on the on-call sheet).

A screenshot below shows what this typically looks like:

New and Permanent Conditions

ICARUS Shift Help group (Added 24 September 2020, and Moved 21 October 2020)

We now have a small team of people who are going to be helping with shifter interactions. Especially this relates to questions about connections as shifters are setting up to take the shift or during the shift. But it could also be useful if a shifter maybe isn't sure where to look for some info and has some general question, e.g. where to look/who to contact.

This team right now is Wenqiang Gu, Francesco Poppi, and myself.

To try to ease these interactions, we formed a slack group including the 3 of us. So in the cases as mentioned above, instead of doing

shifters can start to use the tag
to reach us together.

If you need to restart OnMon (Added 25 May, 2020 and Moved 28 September 2020)

If you need to restart the OnMon, please make sure to close the open OnMon terminal first! Multiple instances open isn't desired.

Current Version of sbndaq is v0_07_01 (Updated 27 November 2020)
We have been running with version 0_07_01 of sbndaq. The icons are mostly in the Desktop folder. It's hopefully open on the DAQ VNC desktop, but if it isn't then you can navigate to it again using the "Caja" program that is in the program menu.