Project

General

Profile

Shift bulletin board » History » Version 186

« Previous - Version 186/270 (diff) - Next » - Current version
Bruce Howard, 01/04/2021 01:48 PM


Shift bulletin board

Don't forget to read both the TEMPORARY and PERMANENT conditions

Colour guide for this page

Current Running Conditions Temporary Conditions New and Permanent Conditions Attention

General comments from the Run Coordinators

Role Name Until (Date) Email Phone Number
Commissioning coordinator Angela Fava January 1st, 2021 8am afava at fnal.gov (xxx) xxx-xxx
Deputy commissioning Coordinator Bruce Howard January 1st, 2021 8am howard at fnal.gov (xxx) xxx-xxx
Run coordinator Name Surname January 1st, 2022 8am someone at something (xxx) xxx-xxx


ICARUS DAQ Expert (also there is the tag @sbn_daq_experts on your post on Slack)

Expert for the week of: 8am FNAL time on January 4 to 8am FNAL time on January 11

Wes Ketchum (@Wes on Slack)

You may also contact all experts on Slack via @sbn_daq_experts.


ICARUS Shift Help Team (tag @icarus-shift-help on your post on Slack)

Experts for the week of: 8am FNAL time on December 7 to 8am FNAL time on December 14

Day / Swing: Bruce Howard / Wenqiang Gu
Night shift: Francesco Poppi


RESUME CALLING FERMILAB MAIN CONTROL ROOM AT THE START OF YOUR SHIFTS!! (Added 23 November 2020)

Shifters should resume calling the Fermilab main control room when they fill out their start shift form. The number to call is in the Start Shift form.

IF YOU (THE SHIFTER) IS CALLED BY MAIN CONTROL ROOM DURING YOUR SHIFT... (Added 23 November 2020)

Then please write a general ECL entry logging the details of the call, and send this ECL entry as an email to Angela Fava, Bruce Howard, and Donatella Torretta.

Google Voice Shifter account (Added 22 December 2020) - Under Test
Shifters now have access to a Google Voice account in order to make/receive calls.
Please see https://cdcvs.fnal.gov/redmine/projects/icarus-operations/wiki/What_to_do_on_shift for Account name and ECL for password.
Remember to log in before your shift and to log out once the following shifter has checked in.


Run EventBuilder PDU check script from terminal in Slow Controls machine (updated 15 December 2020)

Instructions starting 15 December 2020 for the time being.

Instructions
From a terminal in the Slow Controls VNC, run the command

/home/nfs/howard/pdu_check/runCheckEvbPdu.sh
and report its result in the ECL. A suggested title would be "Event Builder PDU Check"

Specifically, you want to check the results of the lines EVB rack, PDU Bank 1: and EVB rack, PDU Bank 2:

This should be run every 2 hours.

If the value reported in the line for Bank 1 is > 20 A or the value reported in the line for Bank 2 > 2 A, then call and alert the Commissioning Coordinator (Angela Fava) or Deputy (Bruce Howard), whose numbers can be found on the Expert Call List.


Use boot_split.txt for DAQ BOOT Configuration until further notice (added 4 December 2020)

Use boot_split.txt for the BOOT configuration in the DAQ Run Control until further notice. This replaces the "usual" boot.txt -- shifters: this applies to both standard shifter runs and lifetime/purity runs.


Restart Event Display at the start of EVERY shift (added 12 November 2020, updated last 10 December 2020)

Reminder to shifters to close any open Event Display and re-open it using the icon on the desktop. This should be done at the start of EVERY shift until further notice. The wiki on the event display is given here and some suggestions for settings are in this ECL entry, see the comment of this ECL post as well. This is a useful tool for looking at events in the cryostats!

NOTE on 10 DECEMBER 2020: Really we want it to be restarting more frequently than every 8 hours. Wes has set up the script to automatically restart the event display every 2 hours instead. We apologize for this inconvenience, but for now this really needs to restart to clear out it's memory. It's important.


OnMon updates in progress (added 10 December 2020)

A test version of reference plots has been added to the online monitoring plots. PLEASE NOTE: there seems to be some mirroring in the reference plots at the moment.

If Grafana is showing data disk(s) >=95% full (added 13 November 2020)

If Grafana shows any of the data disks at or above 95% full (visible near the lower right of the Shifter Grafana page), then please call Wes K, who can be found on the Experts On Call document.


TPC Power Supply checks every two hours (updated 26 December 2020, expert on-call updated as needed)

We're requesting for now that a TPC Power Supply check be done every 2 hours. Basically you run the script

~icarusdcs/sbndcs/icarus/bin/runCheckTpcps.sh

from the terminal in the Slow Controls VNC (or if the terminal that had been running it isn't there for some reason you can open a new one). Then take a screenshot and post in the ECL as a general post with the DCS, Monitoring, and TPC tags]

Best case is it says there are 0 complaints. If there are tiny temperature complaints these are usually okay. If there are temperature warnings > 5 degrees from reference value or > 50 degrees, or for all other warnings please write a message in the shift Slack channel and tag

@icarus-tpcps

If there is no response to Slack after about 15 minutes, then call an expert. If any temperature is >51 degrees or there are warnings on the 12V line (i.e. icarus_tpcps_*_p12), then call an expert.

The primary on-call expert for this week is Mattew Worcester, whose number is in the Experts On Call document and you can also get clicking on "Who is on shift now" in the ECL page side menu. In the case where someone has trouble reaching the primary on-call expert (i.e. they are not answering or responding), then call down the list of TPC power supply experts to attempt to reach an expert.


IMPORTANT!! Ongoing issues with East cryostat pressures & note about who to call (updated 4 January 2021)

Issues with a re-condenser in the EAST module means the pressure may be higher than usual. This is known to the experts.

For the East only, the higher limit to call Claudio is if it goes above 150 mbar relative. For the WEST module, take 140 mbar relative as an upper limit beyond which to call Claudio.

Call Claudio before the Cryo Coordinator if the module's pressure goes out of range. For all other alarms the rule does not change about which expert to call.


Live Event Display (added 21 October 2020)

See the Live Event Display Wiki Page for instructions. Note in particular the "Notes For Usage" passage explaining some important details about using the event display. Also see this ECL entry for some suggested configuration information.


Shifters are encouraged to put the summary of their shift and info passed to next shifter in their shift check-out form! (added 21 October 2020)

The shift check-out form has a box for general comments. Generally, shifters are encouraged to enter the summary they gave to incoming shifter and any notes there as well, and not just in Slack.


CURRENT DAQ CONFIGURATION (updated 10 December 2020)

New conditions for the run with respect to previously:
  • Uses the icons from the Desktop folder on the DAQ VNC (if it's closed for some reason, open the program Caja under the ICARUS program menu and then make sure you are already in or go to the Desktop area in the file browser that opens)
  • Uses the standard configuration and boot.txt FOR NOW WE'RE USING boot_split.txt boot file
  • Use ONLY WE and WW TPC components (icarustpcwe* and icaruxtpcww*), the following 4 CRT components (icaruscrt01nni, icaruscrt01nno, icaruscrt03wci, and icaruscrt03wco), and all PMTs. NO other components

Shifters should go back to filling out the DAQ Checklist and monitoring Grafana and OnMon.

Note that the CRT components are newer to be added to the shifter runs! There is a CRT data rate plot on the shifter page in Grafana. If you notice sudden changes in this rate, please make a General post on the ECL and attach a screenshot and words with what you noticed! Send the ECL form as an email to Antoni Aduszkewicz, Tyler Boone, and Biswaranjan Behera. This is possible by using the "Email new entry to" box. Type part or all of a name and click their name and "add ->" to add each person to the form.

If the run crashes, shifters should go through the usual steps of trying a restart first, but if runs are also crashing after attempting 3 restarts and/or you are having trouble after struggling with the DAQ for 15-20 minutes, then communicate with a DAQ expert (see the current on-call and Slack information at the top of the Bulletin Board).


IMPORTANT!! Changes to iFix instructions for shifters (added 28 August 2020, updated slightly 2 November 2020)

To check for alarms in the system, shifters should look at the Alarm Summary in the top right of the iFix window as well. This shows two types of alarms, regular and critical. For regular alarms the bottom box turns red. For CRITICAL alarms, the top box turns red. See picture below. On alerting cryo experts:

CRITICAL Alarms: call cryo expert whenever this happens (any hour of day or day of week)
Regular Alarms: call cryo expert between the hours 08:00 and 17:00 Fermilab time. If your shift is overnight, pass word on to the next shifters to call when it is again between these hours, so long as the alarm doesn't become CRITICAL.

If pressures are outside the OKAY ranges, call a cryo expert any time of day or day of the week! (NOTE: the instructions were recently updated to call Claudio first in these situations)

Any time you see any alarm make a separate post in the ECL about the alarm noticed and any actions taken.

iFix Alarm Summary


Current Running Conditions

Commissioning Run Schedule (Added 19 August, 2020)

To better plan around activities related to commissioning tests and to help shifters be aware of what activities might be happening, there is now a spreadsheet


Temporary Conditions

If a run is going and suddenly the GUI becomes disconnected! (Added 29 July 2020)

We are trying to keep track of a recent issue we've noticed where a run is going and suddenly the Run Control GUI looks blank and as if it wasn't started (the only button to click on is "Start DAQInterface"). In these scenarios, the run is probably still going and just the GUI has lost connection to the run. We are trying to understand these more.

At the moment, recovering from these are an expert activity. If this happens, please get in touch with Bruce Howard or Wes Ketchum (on the on-call sheet).

A screenshot below shows what this typically looks like:

New and Permanent Conditions

ICARUS Shift Help group (Added 24 September 2020, and Moved 21 October 2020)

We now have a small team of people who are going to be helping with shifter interactions. Especially this relates to questions about connections as shifters are setting up to take the shift or during the shift. But it could also be useful if a shifter maybe isn't sure where to look for some info and has some general question, e.g. where to look/who to contact.

This team right now is Wenqiang Gu, Francesco Poppi, and myself.

To try to ease these interactions, we formed a slack group including the 3 of us. So in the cases as mentioned above, instead of doing

@Bruce
shifters can start to use the tag
@icarus-shift-help
to reach us together.

If you need to restart OnMon (Added 25 May, 2020 and Moved 28 September 2020)

If you need to restart the OnMon, please make sure to close the open OnMon terminal first! Multiple instances open isn't desired.

Current Version of sbndaq is v0_07_01 (Updated 27 November 2020)
We have been running with version 0_07_01 of sbndaq. The icons are mostly in the Desktop folder. It's hopefully open on the DAQ VNC desktop, but if it isn't then you can navigate to it again using the "Caja" program that is in the program menu.