NDOS Commissioning Meeting 2010/12/16

Attendance: Jon, Mark Messier, Mat, Andrew, Phil, Zelimir, Minerba, Tricia, Kurt, Alena, Leon, Gavin, Rick K., Sue, Athans

  1. Shifters' report
  2. DCS Status & Plans
  3. DAQ/Electronics Update & Plans
  4. AOB

Shifters' report:

  • Phil:
    • The DAQ seems to be getting better over time
    • Sometimes during the run we start seeing no data, only u-slice headers.
      • need to do a system reset
      • should first correlate this with plots in Ganglia
    • DDS has crashed once or twice; if that happens, restart DDS and remember to restart the beam spill server
    • Wrote some code to analyze the DSO data that was collected during the week
      • produced a list of dead FEBs, some which have been addressed/fixed (eg, bad cables, loose connections, etc.)
      • some pixel pairs are consistently dead
      • found many noisy FEBs
    • A DSO run was done with cooled FEBs
      • a large fraction of channels develop a 200 kHz noise
      • no obvious pattern in electronics-space
      • need to map these channels into physical locations to look for a patter


  • Chad: not present, but reported previously to Jon that the rack-protection systems are now being read out in Labview
  • Athans: made some progress on sender/receiver applications; will soon add actual code to read back status bits from FEBs


  • Peter:
    • The spillserver had been dying after a very short time. Qiming found that a bit was being reset to 0 in the TDU; a work-around has fixed this problem for now.
    • We are not able to access the status bits of the FEBs while collecting normal data.
    • Ron and Rick are working on the TDU, as is Neil who just got back.
    • The DAQ redmine is now accessible via the "nova" account (with the usual password)
    • We'd like to try turning on the MB triggers
  • Leon:
    • FEBs can get out of sync, so shifters should be looking at the u-slice rates in Ganglia
    • Sue commented that this usually happens at the beginning of a run, so we should pay particular attention to the rates when starting a run.
    • After beginning a new run, check that the u-slice rate is ok (should be ~20 kHz). If bad, do a "restartSystem"
    • Plots to look at on Ganglia:
      • u-slice rate
      • DCM CPU usage
      • corrupt u-slice rate
      • missing u-slice rate
  • Andrew would like to:
    • repeat DSO sutdies with lights out in the NDSB
    • test auto-resynch. After some discussion, it was generally felt that auto-resynch is potentially dangerous, so this will not be implemented.


  • It was noted that some people are not making entries into the CRL when they are doing work out at the NDSB. The general feeling is that we need better communication between people working in NDSB and the Control Room.
  • Mark Messier presented an analysis of recent data that shows a peak of neutrino-like events occurring between 219-223 us. See doc-dB 5452 for details.