NDOS Commissioning Meeting 2011/07/25

Attendance: Jon, Peter, Kurt, Minerba, Susan, Jan, Nick, Andrew, Mat, Martin, Sue, Leon


  1. Beam & Detector Update
  2. DAQ Status & Plans
  3. DCS Status & Plans
  4. AOB

Beam & NDOS Update

  • BNB beam: currently down due to an air conditioning problem. Should be back later today.
  • NuMI beam: problem with power feed for the V108 magnet. 6 out of 16 cables have shorted together. The cables will need to be cut and new cables pulled. This will take several days to repair, and the repair plan is being formulated as of this morning.
  • Friday evening dcm-3-03-01 seemed to fail, however it took a very long time to recognize the problem due to a red herring related to the TDUs.
    • Andrew: there were multiple problems. The TDU is not keeping a GPS lock. Andrew has verified this. Swapped to a different TDU (switched to using. Still saw same error message, but this was a red herring.
    • Andrew: LCC had problems due to lightning strike Friday night. Sat. morning, the entire cluster had to be physically turned on.
    • Peter: overnight (Friday) shift was cancelled.
    • Andrew: 8 am, it was discovered that machines were turned off. Andrew:
      • rebooted cluster remotely
      • physically moved timing chain in MINOS building. Introduces a slave, and should change timing by a few hundred ns, but it appears to be functioning correctly.
      • went to CR and tried to get DAQ running.
      • eventually had to remove DCM 3-1.
      • believes TDU at NDOS is having a problem (hardware).
    • Jon: have the engineers been able to look at the TDU? Andrew: not yet, their talking to GPS unit vendor first.
    • Peter: even on Friday night the TDU looked ok (looking at timing command history buffer), it was synching correctly at the start of the run.
  • Jon: what's the plan for dcm-3-03-01? Andrew: nothing obviously physically wrong. Also, pedestal runner seems to be broken. Andrew managed to read out individual pixels in DSO mode. It's not clear exactly what happens.
    • Sue: it looks like it gets to the "Configuration Hardware" stage ok. It looks like it recognized that it could not read out data and went into an error state. Sounds like this was not being handled appropriately.
  • Peter: real issue is that there was a flood of event megapool error messages. Has asked DAQ experts to address this (eg, Ron's megapool message is not dcm-specific). Ganglia was also down for a while which slowed down progress in identifying the problem.
  • Mark: possible to create a clickable timeline that one can use to track down messages?
  • Sue: it would be good to have a timestamp associated with "close DDS topic reader connection" messages.
  • Kurt: there is a message archive that has all messages that one can use to track down history.
  • Mat:
    • believes he has a setup that's useful. On Friday his system was capable of holding vacuum for several hours.
    • has looked at 11 modules (1 was random, 1 had oil inside o-ring, others had moisture on APDs). Mostly on top horizontal row of diblock 2. APDs have already been removed. Will continue to test today.
  • Jon: results of extended DSO scans? Susan: not done yet. Jon & Susan: probably due to fact that PedestalScanner is not working at the moment. Jon will discuss with Denis.
  • Mark: have we gotten any APDs back from CalTech? Leon: current hold up is that Rick wants tests done after holding APDs at voltage for extended period of time (several-to-24 hours). Will ask Bill Gilbert to take some time to make some changes to the Labview software to allow for these extended tests.
  • Jon: will there be any mechanical changes made to the APD fixtures? Leon's APD meeting begins tomorrow.
  • Mat: there are ~15 APDs at FNAL that could be installed and run warm.
  • After some discussion, we will hold off on installing these until after tomorrow/early Wed. when the APD task force completes their first meeting.

DAQ Status & Plans

  • Timing (Andrew):
    • plots in the logbook show timing over the weekend. Plot on left is time difference between when signal was created vs. when we receive it. Left of peak is completely unphysical.
    • Andrew will continue to analyze the accelerator 1 Hz signal.
    • Andrew: clocks look to be drifting.
    • Peter: would be good to compare to Chad's plots.
    • Jon: what is the plan? Andrew: try to get time-series plots out to help with understanding the problem. Then talk to engineers.
    • Peter: what is the mumbling that's going on regarding putting a direct signal into DCM? Andrew: something that has been discussed as a way to sync DCMs. Jon: is there a request to add this connection to the DCM? Andrew: that's something that the electronics group needs to discuss. Neil, Greg, Rick and Vince will make this request.
    • Peter: Chad and Minerba were going to analyze data to produce fine-detailed delta-t vs. time plot.
    • Mark: it would be good to compare list of runs with bad drift.
    • Sue: has somebody made a plot of MINOS spill time vs. NOvA spill time? Peter: it's been done in the past, and it was shown to be rock-solid. We should also get the data directly from Phil A. (which comes from ACNet) for comparison to what we get from the MINOS spills database.
  • Kurt: might have some CfgMgr changes implemented later this week.
  • Andrew: would like to swap TDU hardware and have someone else besides Denis and Andrew to bring the system up.
  • Andrew & Peter: we need more help from the collaboration on the DAQ, in particular on-call experts. This would be a good place for a new post-doc to help out.
  • Leon: TEC remaining noise problem needs further work later this week.

DCS Status & Plans

  • Martin: now displaying TEC drive current and FEB temperature. Just raw data is being displayed for now.
  • Next DCS meeting is tomorrow morning 11 am in the Darkside (WH 6X)


  • Andrew: can we get coffee pot on 12th floor?
  • Andrew: new shifters were not able to fully describe the extent of the problem. We need formal training. Jon: will try to put something together need specifics. Peter: also need a shifter report. Mark: rookie's guide from Young NOvA would be very helpful.
  • Mark: would be nice to reformat wiki to make it easier to navigate.