Shift bulletin board » History » Version 89

« Previous - Version 89/116 (diff) - Next » - Current version
Robert Wilson, 09/01/2020 08:50 AM
Remove "NEWLY ADDED !!!" for an entry over a month old

Shift bulletin board

Don't forget to read both the TEMPORARY and PERMANENT conditions

Colour guide for this page

Current Running Conditions Temporary Conditions New and Permanent Conditions Attention

General comments from the Run Coordinators

Role Name Until (Date) Email Phone Number
Commissioning coordinator Angela Fava January 1st, 2021 8am afava at (xxx) xxx-xxx
Deputy commissioning Coordinator Bruce Howard January 1st, 2021 8am howard at (xxx) xxx-xxx
Run coordinator Name Surname January 1st, 2022 8am someone at something (xxx) xxx-xxx

IMPORTANT!! Changes to iFix instructions for shifters (added 28 August 2020)

To check for alarms in the system, shifters should look at the Alarm Summary in the top right of the iFix window as well. This shows two types of alarms, regular and critical. For regular alarms the bottom box turns red. For CRITICAL alarms, the top box turns red. See picture below. On alerting cryo experts:

CRITICAL Alarms: call cryo expert whenever this happens (any hour of day or day of week)
Regular Alarms: call cryo expert between the hours 08:00 and 17:00 Fermilab time. If your shift is overnight, pass word on to the next shifters to call when it is again between these hours, so long as the alarm doesn't become CRITICAL.

If pressures are outside the OKAY ranges, call a cryo expert any time of day or day of the week!

Any time you see any alarm make a separate post in the ECL about the alarm noticed and any actions taken.

iFix Alarm Summary

IMPORTANT!! Ongoing issues with East cryostat pressures (updated 20 August 2020)

Issues with a re-condenser in the East module means the pressure may be higher than usual. This is known to the experts. The set point for overpressure condition on the EAST module at which one should call Claudio is now 180 mbar relative (until further notice). For the WEST module there are no changes. Thanks.

Shifters checking Online Monitor to see if a portion or entirety of a particular mini-crate is having issues: (updated 21 August 2020)

Since the instructions are still at a temporary stage and need improvement, the instructions and a few relevant pictures are on this ECL post . If you notice a problem with a board or mini-crate, there are instructions there as well. This can be with any mini-crate too, see for example the screenshot in this other ECL post for how something looked in a different instance on a different mini-crate.

PMT HV for some channels is ON. Shifters are to monitor HV and fill out related ECL Form (Added 11 July 2020)

Some PMTs' HV are now ON! See the Wiki page at PMTs On Or Off Page or relevant posts in ECL (e.g. 8405).

Shifters are to keep watch using the PMT display(s) in the DCS (Slow Controls) VNC [where you may have previously monitored the LAr fill levels] and fill out PMT HV Monitoring form on the ECL every hour. See the Wiki page on PMT HV Monitoring for details!

Old cryogenics form replaced with new form! (Added 23 June 2020)

Instead of filling out the "Cryogenics Checklist" we now ask shifters to fill out the "Cryogenics Stability Checklist," as we are now beyond the fill. Shifters should keep the same schedule of Cryo forms, at least for now, just using this new form instead.

NEW FORM for day shifters! (Added 4 June 2020)

As noted in the What To Do on Shift page, there is a new form that day shifters are to fill out just after starting their day shift. The form will lead you to a spreadsheet (link in form and in the Projects tab in ECL) which will lay out a schedule of planned work at the ICARUS building. We want to have a form each day for people to easily look at that collects the information for the given day. Your help is greatly appreciated!

When Form Category Tag Instructions
Day Shifters right after start of shift Daily ICARUS Access General Access, Work Plan

Clarification added 6 June 2020: If there is no access planned on a weekday, a form should still be filled out indicating that. During the weekend, if something isn't listed in the spreadsheet for a day, it's probably okay to skip the form. But if you want to still fill one out indicating there is nothing you know to be planned, you can still do this!


Effective 18 March 2020 single cryo expert expert entry is no longer a shift-taker responsibility

Current Running Conditions

Commissioning Run Schedule (Added 19 August, 2020)

To better plan around activities related to commissioning tests and to help shifters be aware of what activities might be happening, there is now a spreadsheet

Issue with the "All Channels" notification in PMT HV monitoring overview page (Added 7 August, 2020)

On the PMT HV monitoring overview page, there is a known issue where the Eastside PMT "All channels" are showing off, but it is actually ON. (DO NOT CLICK OR DO THINGS WITH THIS BUTTON).

The individual channels on the individual channel monitors show this ON status. Experts will work to fix this but in the meantime the individual channels will show you the channel statuses.

Note on current status of OnMon (Updated 29 July, 2020)

OnMon has been updated. The wiki page reflects the changes from the shifter's perspective! See the OnMon wiki page.

If you need to restart OnMon (Added 25 May, 2020)

If you need to restart the OnMon, please make sure to close the open OnMon terminal first! Multiple instances open isn't desired.

Components to use in data-taking (Added 3 April, 2020)

We have done some testing of PMT running with the West-side mini-crate runs and we will now start including most PMT components in the west-side mini-crate runs. So, runs on the west-side mini-crates (the typical ones started by shifters) should include:

WE 05, 06, 07, 09, 18, 19

WW bot 01, 02, 03
WW top 01, 02, 03
WE bot 01, 02, 03
WE top 01, 02, 03
EW bot 01, 02, 03
EW top 02, 03 (explicitly NOT EW top 01)
EE bot 01, 02, 03
EE top 01, 02, 03

Note that this is all the PMTs except for EW top 01, which runs on the SPEXI setup used by the east mini-crates.

For the expert-started runs, these will use TPC EW 05, 06, 07, 09 and PMT EW top 01.

Taking runs during the cool down [and early commissioning period] (Updated 3 April, 2020, title updated 29 June 2020)

We're now piloting pseudo-continuous running of the DAQ! We request that a good effort be made to keep runs going with Online Monitoring for the West side mini-crates (WE 05, 06, 07, 08, 18, 19) [ Update 3 April the above entry here notes that we are also now going to run PMTs when we take the west side data ]. Please make an ECL entry every 2 hours with the run number, the details of the run [in Grafana the data rates from the gauges, the alarm statuses, etc.], if Online Monitoring is present, and if any issues were encountered. During this period, do NOT call experts if the DAQ or OnMon isn't working. Make an ECL post explaining the situation, and if you want, you can post in the #daq or #operations channel of the SBN Slack in case someone happens to be looking. While we want to take noise data consistently, we do not want to overburden experts during this period.

For more details, see the page describing how to take a run, and the page on Online Monitoring for the shifter.

If there is an issue with the run (something seems wrong, there are alarms in Grafana, etc.) then you should make note of this in the ECL. You should feel free to add screen shots to the ECL post. Bring down the run (all the way to STOPPED using Terminate after Stop), close the OnMon terminal, restart OnMon and then restart the run. When the new run starts, make an ECL post again with the details of this run.

If OnMon stops updating, please close the Online Monitoring terminal and restart OnMon. You may see red messages in the Message Viewer for a few minutes while the OnMon restarts. Give it several minutes (say 10 minutes) to restart before declaring there is a problem with the run and restarting following the instructions in the preceding paragraph.

Experts who wish to do work/testing with the DAQ that could affect the shifter's data-taking should coordinate with the shifters before doing said activities. Write everything in the ECL

The following is an example ECL post to update on the status of a run at the 2-hour check-in points. Starting 14 March 2020, we have the DAQ (Commisioning) Checklist which has the necessary elements to fill out:

One thing we just thought to add: please also note on this "every 2 hours" post if the Archiver is running (this is in the OnMon pages on the bottom right monitor of CR 01)

Connected Mini-Crates (Updated 14 February, 2020)

Connected: Ten crates are currently connected: EW 05, 06, 07, 09 and WE 05, 06, 07, 09, 18, 19. None of the VST optical fibers are being used.

Fragment IDs: The channel number assignment is based on the fragment ID FHiCL parameter. The channel numbers occur in blocks of 576 channels offset based on the fragment ID (e.g. fragment ID = 0 corresponds to 0-575, fragment ID = 1 corresponds to 576-1151, etc...). The current mini-crate(fragment ID) pairs are: WE05 (0), WE06 (6), WE07 (1), WE09 (2), WE18 (8), WE19 (9), EW05 (3), EW06 (7), EW07 (4), EW09 (5).

Triggers: The WE crates are still hooked up to receive triggers using the pulser to generate a TTLink signal, however for testing the trigger board, the EW crates are now getting the TTLink from the trigger board via the east-side fanout. See the related ECL entry

The TTLink out from the trigger crate (WE05) is currently not being sent to the oscilloscope before going to the TTLink fan-out modules. See ECL entry 1107.

For now, running should include only one set, the WE mini-crates OR the EW mini-crates.

Trigger Conditions (Updated 17 January 2020)
The Tektronics Pulser AFG3252C is being used to simultaneously send a pulse to the TPC and PMT racks. Channels A (TPC) and B (PMT) are synchronized. The settings for both are:

  • Freq 5 hz
  • Delay 0 ns
  • High 3 V
  • Low 0 V
  • Offset 1.5 V
  • Width 2 micro seconds.
  • Leading 9 ns, trailing 9 ns
  • Freq 5 hz
  • Delay 0 ns
  • High 5 V
  • Low 0 V
  • Offset 2.4 V
  • Width 2 micro seconds.
  • Leading 9 ns, trailing 9 ns

Howard B. previously pointed out (before December 26) that having the AFG3252 pulser (old SBND borrowed pulser) with a 200 micro seconds window caused the TTLink signal to be 63 Hz after power cycling, which is not right as it should be 10 Hz. However, having the new pulser at 2 micro seconds fixed the problem.

N.B. (31 January 2020) four crates (WE minicrates) still use this configuration and four (EW minicrates) use the trigger crate signals for testing purposes (see the "Connected Mini-Crates" above)

Temporary Conditions

If a run is going and suddenly the GUI becomes disconnected! (Added 29 July 2020)

We are trying to keep track of a recent issue we've noticed where a run is going and suddenly the Run Control GUI looks blank and as if it wasn't started (the only button to click on is "Start DAQInterface"). In these scenarios, the run is probably still going and just the GUI has lost connection to the run. We are trying to understand these more.

At the moment, recovering from these are an expert activity. If this happens, please get in touch with Bruce Howard or Wes Ketchum (on the on-call sheet).

A screenshot below shows what this typically looks like:

New and Permanent Conditions

Current Version of sbndaq is v0_06_01 (Updated 2 June 2020)
We have been running with version 0_06_01 of sbndaq, using the icons in the folder sbndaq-v0_06_01 on the VNC desktop! It should be open, but if not, then open it by double clicking on the folder icon.