Previous Page Main Page What to do while on shift DAQ Trouble Shooting Guide

How to know whether a run is producing (usable) data

Updated 2011-12-29



  • Look at the Cold Start How-to to learn more about some of the tools and apps mentioned here.
  • See also the DAQ Trouble Shooting Guide for more information, although that has older (and stricter) requirements on DCM Microslice Rates. Please follow the guidelines below in the current page.

Are the DCMs synced?

See this link.

(The older version is here.)

Return to top of page

Are the Trigger Scalars updating?

You should see cosmic, NuMI and BNB trigger counts increasing steadily.

  • If Calibration triggers aren't counting, then there is a problem that probably requires an expert.
  • If the TriggerScalars aren't updating new spills every few seconds then:
    • Verify that the beam is on ( check Channel 13 )
    • Attempt to resart the spill server:
    • If you are still not receiving spill triggers after a few tries call an expert

Return to top of page

What's seen on the Memory Viewer?

  • Nothing is changing at all in the Memory Viewer?
    • Sounds like the DAQ is not producing any events, even empty ones. This can happen temporarily at the beginning of a run, particularly when the detector hasn't been synced yet.
    • If the memory viewer remains static after a half a minute or so, there's probably something seriously messed up with the run.
  • Some data in the Memory Viewer is updating, but the event is always the same size.
    • If the Memory Viewer just shows a few lines of changing numbers, but the size never changes, and there aren't triplets of white, light-green, and dark-green boxes, the DAQ is producing empty events. This is usually because one or more DCMs is not producing data in time to be found by the trigger in the buffer nodes.
    • This can often be cured with issuing a new "Sync Detector to Current Time".
    • Sometimes, even when the latter works, it can take a minute or more for the DCMs to catch up. It's possible that the DAQ will start producing good data a minute or two after a sync.
  • Lots of hit data (white/light-green/dark-green triplets) in the Memory Viewer, but big chunks of gray in between.
    • This usually means one or more DCMs is producing fragmented or corrupted data.
    • Try a resynch, or restart the run.
  • Lots of hit data (white/light-green/dark-green triplets) in the Memory Viewer, no big gray chunks mixed in (except in the first line
    or two, and a few words here and there for microslice headers).
    • This is probably a good run. Here's an example:

Return to top of page


  • The Ganglia web pages (usually viewed on nova-daq-03) have a lot of information about the system, particularly about the health of the data structure produced by the DCMs. See the Ganglia Wiki page for more information. Pay particular attention to
  • Microslice rate
  • Microslice corruption rate
  • Microslice size

Return to top of page


The Event Display currently shows a lot of noise, but it shouldn't take more than a few seconds to see a muon correlated in both views. If you don't, there might be a synchronization or other problem. Resync or restart...

Return to top of page


The Online Monitoring hit maps should show something in all DCMs. If not, you'll probably need to restart the run.

Return to top of page


All the boxes should be green as shown. See here for full information: All_about_DAQApplicationManager.

Previous Page Main Page What to do while on shift DAQ Trouble Shooting Guide