Project

General

Profile

RC - Overview

Brief description of the DAQ

The data flow

We have 9 TPC readout crates and one PMT+Trigger readout crate. Each crate is connected to a PC (called an SEB) that configures, monitors, and receives data from the crate. Luckily for us, the crate numbering scheme is the same as the DAQ PC: TPC crate 1 --> SEB01, crate 2 --> SEB02, etc. SEB10 handles the trigger/PMT crate.

When we trigger, data is pushed from the readout crates to the SEBs. When enough data has been collected, data on the SEB is processed to find where the data from one event are, and that data is sent over a local network to the event builder PC (EVB, or assembler). The event builder collects all of these fragments, and waits until it receives one from every event, and then it writes that event to disk.

The run control

Our run control program is called "runConsoleDAQ". It determines the type and length of the run (or, we tell it!), it launches all the necessary processes, issues commands to them to configure/start the run/stop the run/etc., and does some basic monitoring of the processes to know what state they are in and if they are running. It is largely automated, including making automated elog entries at the beginning and end of runs, though it can require some user-intervention and occasionally has some problems, so it's good to keep a close eye on it.

What you're looking at when you look at the DAQ VNC

Instructions for bringing up the DAQ VNC and starting a run are all located in the RC - Guide. That's a very good thing to read through! When you look at the VNC, usually it will look like this below:

  • The black terminal with white text on the right (which is logged in to the EVB) is the "runConsoleDAQ" terminal. That's where we run the run control from. If we haven't started a run, that will likely be the only terminal open.
  • The large blue window on the left is the assemblerApp process, running on the EVB and collecting and writing all the data.
  • The smaller blue windows in the middle are the sebApp processes: one each on the SEBs. These are what collect the data from the readout electronics, and send it to the EVB.
  • The bottom two black windows with orange text on the left are the programs from running the online-monitor: the dispatcher, on the left, sends data from the EVB to the online-monitor process; the online-monitor, right in the middle, gets that data and analyzes it, making the plots you see running on the lizard online-monitor.
  • When we start a run, a "hardware configuration" routine runs, which may be short or long depending on what exactly is configured. This appears as a small black terminal with green text on the left (between where the assembler and dispatcher windows are). Once finished, it disappears.

At the beginning of every run, the runConsoleDAQ starts all the processes via xterms, so you will see these terminals pop up. At the end of the run, it cleans them all up before starting a new one, so you should see them disappear.

The Ganglia monitoring

We can watch internal workings of the DAQ processes using the ganglia web-based monitoring program. Instructions for opening the page are part of the guide. The plots shown in the shifter view are:
  • Event Builder: Event Rates: this is the rate of assembled (blue) and written (red) events on the assembler. The writing usually exhibits more spikey behavior, but overall these should roughly agree.
  • Event Builder: Dropped Events: this is the number of dropped events in the assembler process. That can happen if we are taking too long to write the data, or have too high a data rate. We want this to be zero!
  • Event Builder: Max Events in Queue: this is the maximum number of events in the queue to be written in a short sampling time. It's often more than one, but shouldn't usually be more than 10. What's important, is that it's stable on average---that means we are keeping up with the rate!
  • ubdaq-prod-evb-priv.fnal.gov Network last hour: this is the network traffic on the EVB. Green is incoming data rate (should be dominated by the SEBs sending data), and purple is outgoing data rate (data going out to tape!).
  • SEBs: Outbound Data: this is the outgoing data rate on each of the SEBs (1-10). Typically, the TPC SEBs (1-9) are all rather similar, while SEB10 sends far less data.
  • SEBs: Outbound Fragmants: this is like the above, except the outgoing fragment rate, which is basically the event rate seen by the SEBs. Unlike the data volume, the event rate should be the same for all SEBs.
  • SEBs: Outbound Queue Occupancy: this is the average number of fragments waiting to be sent to the event builder on each of the SEBs. Typically, it's zero, because fragments move out very quickly, but occasionally you can see it spike up when it catches a fragment there. It shouldn't ever be very high though.
  • SEBs: Circular Buffer Occupancy: this is the amount of data sitting in the SEB buffers waiting to be processed and sent to the event builder. We configure it so a constant amount of data is on each SEB prior to it being sent out. Here, we want to make sure this is stable: that means that data coming in is being matched by data going out!