DDT Monitoring Chain

There are three basic tiers of DDT monitoring:

  1. Process raw files
  2. Create plots
  3. View plots online

While this list is oversimplified, these three levels of operate almost entirely independently of each other. The common thread here is that the output of each tier is the input for the next. The goal here is to produce metrics which permit quick, simple monitoring of critical information for each trigger.

Raw File Processing

Everything begins with the raw files. For DDT, there is a raw file associated with each combination of subrun and active trigger.

These are processed and converted into ROOT files by the OnMon producer module. (This is, despite the name, an art analyzer module.) The output file sizes are much smaller than those of the input raw files, though this is accomplished by discarding information unnecessary for current metrics.

Presently, the vanilla OnMon producer is used to process DDT raw files for monitoring. However, since OnMon wasn't designed with DDT in mind, this loses a lot of trigger-specific data due to the way it handles files from the same subrun. To mitigate this, a DDT variant of the OnMon producer is in the works. This will allow the producer to select different pieces of the raw file to keep and discard, as needed by various DDT metrics. For details on the differences between OnMon and DDT OnMon, please refer to the page dedicated to those changes.

Running the OnMon producer involves a number of involved steps, but those are best explained via the OnMon wiki. For this process as performed by DDT OnMon, please refer to the steps detailed here.

Plot Creation

The plots made here are the core output of DDT monitoring. The DDT nearline plot maker repackages the information contained in the processor's output ROOT file in a more readable format and performs some quick computations as required by individual metrics. There is a code hierarchy here:

  1. Top-level Bash script
  2. Mid-level ROOT script
  3. Bottom-level ROOT script

The purpose of each of these will be changing soon, so I'll hold off on detailing their functions until that point.

The plan is to modularize this step so that metrics can be easily created and added to the list using currently available data sets from the processed ROOT file. This will automate most of the formatting of plots, but it will allow collaborators to easily insert the information contained in the ROOT file's histograms (or relatively simple computations involving them) into graphs or histograms to be displayed online.

There are a number of metrics which generate plots, but all of them follow the same basic pattern:

  1. Get required information from available histograms within the processed ROOT file.
  2. Manipulate this information to obtain the values required for the metric to function properly.
  3. Set up error handling for the metric, which typically involves defining ranges of good values.
  4. Produce the plot and output it to a PNG.

Online Viewing

This is the ultimate goal: quickly viewing the latest, most important information about DDT trigger behavior at any time. This is done via the full list of plots on the Data Quality group's nearline website.

There are scripts associated with the site which periodically check /nusoft/app/web/htdoc/nova/datacheck/nearline/plots/ for updated plots. The DDT nearline plot maker outputs its plots here in PNG form for easy digestion by HTML. When the nearline site refreshes its stock of plots, the latest version of each plot is displayed.

However, this updating can break in many unexpected ways. To assist shifters in recognizing these occurrences, the bottom-left of each plot contains information about when it was last updated. The plots for each time period update at different intervals, but a good rule of thumb is to become concerned if a plot hasn't updated within about 5% of the time period covered.