NDOS Commissioning Meeting 2011/07/21

Attendance: Andrew Norman, Mat Muether, Minerba Betancourt, Peter Shanahan, Susan Lein, Martin Frank, Kurt Biery, Raphael Schroeter, Denis Perevalov, Chad Johnson, Mousumi Datta, Jan, Nick, Dominick, Kaneka?, Vince Pavlicek, Leon Mualem, Mark Messier


  1. Beam & Detector Update
  2. DAQ Status & Plans
  3. DCS Status & Plans
  4. DataCheck

Beam Update

  • nt02 is in place. Scans to begin soon. (Problem with fused quick-connect leads needs to be fixed first.)

NDOS Update

  • Mat has been working on setting up system to test optical connector for leaks. And setting up jig to measure profile of optical connectors and associated hardware.
  • Denis has been working on TECC noise test. Found 5 good TECC noise candidates. Will probably find more. They will be studied more in next week's workshop.
  • Other: Mat reports some as-built measurements of cable trays were going on in the NDOS hall.

DCS Update

  • General (Martin)
    • There'll now be a regular meeting at 11am, Tuesday.
    • There's now a DCS mailing list on listserv.
    • Martin is working on adding drive current and FEB temperature to display in control room. Maybe roll out tomorrow.
    • There will be a discussion at next week's DCS meeting about DCS states.
    • Will also hear update (in DCS meeting) about Alarm work by Gennadiy.
  • NDOS Hall Dewpoint (Chad)
    • Dewpoint has been creeping up close to 10C. This is not far below cooling water tempearture, so there's a concern about condensation on the heat-sinks leading to FEB corrosion.
    • We don't have a clear quantitative analysis about the headroom needed between the dewpoint measured in any one location and the water temperature, but this is getting close on the scale of variations in dewpoint measured between several locations.
    • Mat & Chad will bump the cooling water temperature a couple degrees (F) after the meeting.

DAQ Status & Plans

  • TDU drift (Andrew)
    • Continues to investigate. Now have a new TDU firmware version that allows decoding the TCLK pulse-per-second. Also made a debugging copy of SpillServer to log a 4-day long history buffer of all the decoded timing events (NuMI, BNB spills, PPS).
    • Several anomalies have been observed, such as the PPS suddenly moving by 10's of milliseconds.
    • Overnight, the NDOS TDU time seems to have stopped updating, although output clock signals continued to run, but they were probably undisciplined. The number of satellites claimed by the TDU stopped fluctuating (which is unusual) and the GPS lock bit stayed on. Investigation by Greg and Neal found that indeed GPS was not updating. This would have been obvious on the next run synch, which would have led to empty events.
  • Mark is not convinced that all NOvA-BNB spill time issues are on our side. He learned from MiniBooNE DAQ experts that their time could be off, and it would have no effect on their ability to record neutrinos.
    • Peter: a comparison of NOvA:MiniBooNE::NOvA:MINOS for a short time period (say, 1 hour) in which Chad sees NOvA-MiniBooNE drift might tell us whether that effect is due to NOvA or MiniBooNE. ACTION ITEM: Chad and Minerba will work on this.


  • Susan presented docdb 5978 (also Jan, Dominich, Cristian)
  • There are times when circumstances lead to a very large number of FEBs doing an auto-shutoff (due to rate) during a run. There's no easy way for the shifter to recognize this.
    • There is a FEB vs. time plot in OM, but it's not currently filled properly. ACTION ITEM: Susan will remind Alena about this issue.
    • We also need to alarm on such a condition.
      • Mark - there are hooks in onmon_view for this - e.g., turn a histogram red.
      • Peter - communication with message logger is part of the original plan for onmon, so it would be good to have such alarm messsages go to the msglogger. We hesitated because that would introduce an online dependency to onmon, which we also want to run offline (in principle). But no one is using it now offline (because it's an FMWK-based package). And, MessageFacility is known to ART, so when onmon moves to ART there won't be a problem. ACTION ITEM: Peter will investigate what it will take to teach onmon about MessageFacility.
  • NuMI event peak
    • Why are there no entries after the May shutdown?
      • There seems to be an issue with the back-filling of POT data for that period.
      • ACTION ITEM: Susan will check with Jarek.
      • Also, it's not necessary to make a cut on POT in the un-normalized event time vs. run plot.
    • Normalization
      • They've tried normalizing by POT on a sub-run basis.
      • Mark points out that this will tend to enhance background in short runs.
      • Normalizing on a per-spill basis is technically correct, but has the same pitfall for low intensity spills.