NDOS Commissioning Meeting 2011/02/14

Attendance: Jon, Kurt, Paul, Peter, Mark Mathis, Mark Messier, Mat, Gary, Tom, Susan, Ryan, Leon, Minerba, Sue, Jarek, Rick

  1. Channel Recovery Plans
  2. DAQ Update & Plans
  3. DCS Status & Plans
  4. Shifters' report
  5. AOB

Channel recovery:

  • Mat:
    • Last Thursday: 11 new APDs were put on top of diblock 1
    • DSO scan was done and all channels are in readout
  • Leon: A number of channels have never been able to read out; the APDs will be removed from the detector this week
  • Mat did an inventory of missing channels. There is a mix of bad APDs and FEBs, and is working on determining which channels need to be removed/fixed.
  • There will be a meeting tomorrow (2:15 pm Central) to plan.
  • A new box of heatsinks arrived last week (40 in the box).
  • Gary: have we successfully recovered oily channels?
  • Mat: yes. cleaning optical connector and installing new APDs with shims seems to work. Oily APDs have not shown to be recoverable.


  • Peter:
    • On Saturday there was a power glitch that shut down the accelerators and DAQ machines. This resulted in a loss of data until Sunday morning.
    • Some communication issues/finding relevant DAQ experts
  • Jon:
    • after the DAQ machines were rebooted (thanks to Denis) on Saturday, we were unable to get a run started.
    • upon investigation (and locating the appropriate log files), it was discovered that the Data Logger was crashing after it received the "BeginRun" message from Run Control
    • the reason it was crashing was because it was trying to attach to a shared memory segment that was too small.
    • there were some delays in contacting Andrew Norman, but when he was contacted, he realized that the shared memory segment was only 1 kB, and the DL requires ~20 MB. Andrew destroyed and re-created the shared memory segment with a size that we thought was large enough.
    • the shifter then tried to again start a run, but ran into the same problem. Jon realized that the new shared memory segment was not quite 20 MB, and so the DL still crashed.
    • Jon tried to follow up again with Andrew, but was unable to reach him for ~40 minutes. At that point, Jon told the shifter to go home. Two minutes later, Andrew called back. Too late. In the end, Andrew increased the shared memory size to ~40 MB. The day shifter was able to start a run first thing in the morning.
  • Sue: is the 3 hour problem gone?
  • Peter & Jon: looks like it. Possibly some spikes on Friday? Over the weekend, the DAQ ran overnight with no problems.
  • Software updates:
    • Run Control (later today)
    • Data Logger (later today)
    • DCMApp (
    • ExternalPackageTest (Sue has added some utilities
    • basically all packages
  • Kurt: this will pick up the changes to the ConfigurationManagement that will enable (but not by default)
  • Leon: has some code that dumps out thresholds... would like to compare notes with Sue
  • Sue: datamon is showing a memory leak, probably the reason datamon crashed
  • need a system-wide ulimit set (Peter will ask Andrew/Denis to do this)
  • this morning we also had problems starting a run; immediately after starting the run, some DCMs went haywire.
  • DCM status readback:
    • should run some readback executables
    • Sue will contact Qiming and Andrew
    • Sue will run tests on teststand
  • Sue:
    • dcm-08 is running fine, dcm-09 is wigging out (FEBs are out of sync)
    • has been able to reproduce wig-out problem on teststand
  • Peter: do we have a digital scope (3034) that we can hook up an antennae to?
  • Jon will ask MINOS Run Coordinators
  • Mat: rebooting TDU has helped in the past. Will try reboot TDU.


  • Jon will begin working on a DCM DCS monitoring application

Any Other Business: