Project

General

Profile

Know whether a run is producing (usable) data » History » Version 10

« Previous - Version 10/25 (diff) - Next » - Current version
Holger Meyer, 12/29/2010 10:57 PM


How to know whether a run is producing (usable) data

  • Look at the Cold Start How-to to learn more about some of the tools and apps mentioned here.
  • See also the DAQ Trouble Shooting Guide for more information, although that has older (and stricter) requirements on DCM Microslice Rates. Please follow the guidelines below in the current page.

Are the DCMs synced?

See Detector_Sync

What's seen on the Memory Viewer?

  • Nothing is changing at all in the Memory Viewer?
    • Sounds like the DAQ is not producing any events, even empty ones. This can happen temporarily at the beginning of a run, particularly when the detector hasn't been synced yet.
    • If the memory viewer remains static after a half a minute or so, there's probably something seriously messed up with the run.
  • Some data in the Memory Viewer is updating, but the event is always the same size.
    • If the Memory Viewer just shows a few lines of changing numbers, but the size never changes, and there aren't triplets of white, light-green, and dark-green boxes, the DAQ is producing empty events. This is usually because one or more DCMs is not producing data in time to be found by the trigger in the buffer nodes.
    • This can often be cured with issuing a new "Sync Detector to Current Time".
    • Sometimes, even when the latter works, it can take a minute or more for the DCMs to catch up. It's possible that the DAQ will start producing good data a minute or two after a sync.
  • Lots of hit data (white/light-green/dark-green triplets) in the Memory Viewer, but big chunks of gray in between.
    • This usually means on or more DCM is producing fragmented or corrupted data.
    • Try a resynch, or restart the run.
  • Lots of hit data (white/light-green/dark-green triplets) in the Memory Viewer, no big gray chunks mixed in (except in the first line
    or two, and a few words here and there for microslice headers).
    • This is probably a good run. Here's an example:

Ganglia

  • The Ganglia web pages (usually on nova-daq-03) has a lot of information about the system, particularly about the health of the data structure produced by the DCMs.
  • Microslice rate
    • A microslice is a collection of all hits (nanoslices) for all Front End Boards (FEB) on a DCM, for a fixed time interval. Normally, a new microslice is produced every 50 usec. Therefore, the nominal rate is 20 kHz.
    • If a single FEB produces data even slightly out-of-sync from the others, the data from the DCM gets fragmented into multiple microslices for the same time period. This can cause microslice rates of 40, 60, or even 100kHz. If this is the only problem, and the EventDisplay shows a nice, long muon once in a while, there's probably nothing terribly wrong. However, 100 kHz is taxing to the DCM CPU, and the run may fail after a while.
    • If a few resync attempts don't fix a very high microslice rate, you'll probably need to start a new run.
  • Microslice corruption rate
    • Various problems can cause the DCM to label a microslice as corrupt. Many of these are related to the beginning and ending timestamps of the microslice, so there's some correlation with high microslice rates.
    • If a DCM has a 60 kHz microslice rate, and a 20 kHz Corrupt microslice rate is probably taking mostly good data.
    • If the corrupt microslice rate is close to the total microslice rate for a DCM, then it's likely not producing any usable data.
    • A resync can be tried, but starting a new run, going back at least to "Reprepare Hardware" will likely be needed.
    • I have found that the rate of corrupt microslices will be zero with one or two attempts at starting a new run. --HM, 12/29/2010

EventDisplay

The Event Display currently shows a lot of noise, but it shouldn't take more than a few seconds to see a muon correlated in both views. If you don't, there might be a synchronization or other problem. Resync or restart...

Histograms

The Online Monitoring hit maps should show something in all DCMs. If not, you'll probably need to restart the run.