Project

General

Profile

How to Interpret the Nearline Frontage Plots Near Detector

This page is designed to explain to you what the nearline front page plots should look like and cover the main failure modes that you are likely to see in each plot and how to recover from them. If there are any issues with this page, email both and .

Temporary conditions may cause features to be present. Always read the shift bulletin board for the latest running conditions announcements.

General Information

  • First note the timestamp on the bottom left corner of the plot. This should be within the last 15 minutes (or 20 for the GoodRuns plot). If this is not the case, then this indicates that the web-plot making scripts aren't running.
  • The new data appears on the right edge of the plot. Note that Near Detector subruns are ONE HOUR LONG. This plot will only update when a subrun is finished, so it will take ~1.5 hrs to update. If there is a lot of white space on the right side of the plot and the detector is on with a run going, then this is an indication that the nearline processing has stopped.
  • If you see either of these issues, email .

Near Detector Nearline Front Page Checklist Plots

The following are the plots used for the Near Detector nearline checklist. For normal running these plots should show behavior which is constant over time. Any deviation could indicate an issue.

The four plots are Number of Active FEBs per Subrun, the Timing Peak, the Good Subruns and OnMon FEB Hit Rate Spectrum vs. Time. Click on the links to learn more about each one. During beam shutdown, Number of Noise Slice Hits per Subrun and Average Track Length by Subrun will be on the frontpage instead of the Good Subruns and Timing Peak plots.

The plots are described below along with what most failures mean. If you cannot find the solution here, call your DAQ expert.



Number of Noise Slice Hits per Subrun

This plot displays on the frontpage only during the beam shutdown.
It shows the average number of hits in each noise slice in a subrun.

Below is the month version of this plot, showing the expected values during beam shutdown:

This should have an average value between 25 and 35 hits/slice. If you see points above 40 hits/slice, please email .



Average Track Length by Subrun

This plot displays on the frontpage only during the beam shutdown.
It shows the average track length (in cm) for all 3D tracks in a subrun.

Below is the month version of this plot, showing the expected values during beam shutdown:

The points should lie between 365 cm and 375 cm. If you see more than one point in a row below 360 cm or above 380 cm, please email .



Number of Active FEBs per Subrun

This plot shows the number of FEBs (or APDs) that report any hits in a subrun.

It should look like

This is GOOD data.

The dashed green line represents the total possible FEBs that could be active (10,749 for the FarDet, 631 for the NearDet).

The x-axis will auto zoom out so if we were running with some of the detector missing over the last day the x-axis range could be much larger.

Over the course of a subrun we expect that a few (0-2) FEBs will stop reporting (drop out) due to them being too noisy. Once EVERY 10 MINS a recovery message is sent to all channels which will recover these dropped out channels.
  • If you see number of channels decrease over several subruns then AutoStartDAQ might not be running. Check that the indicator next to "Auto StartDAQ" in the TDU Control Interface is green. If it is red, click on the Timing tab and select "Auto startDAQ".

Occasionally, things can cause a large number of channels to drop out at the same time (cell phones, lights).
  • If you see a large drop (greater than 2) channels in one subrun then you can manually recover these channels by issuing an "Enable FEB Data Flow" (green button) from the TDU Control Interface.
  • If you see an entire DCM not reporting in OnMon or the Event Display you may need to issue a “SYNC” (red button) from the TDU Control Interface.

This plot (and all the nearline plots) will take about 1.5 hrs to update. Therefore look in OnMon and the EventDisplay for the missing channels to come back.
If after 10mins OnMon/EventDisplay still show that many channels have dropped out, issue a “SYNC” (red button) from the TDU Control Interface.

If you have issued both an "Enable FEB Data Flow" and a “SYNC” and after 5 mins no improvement is seen in the Event Display or OnMon (remember nearline is more than an hour behind), call your DAQ expert!

Note: If there is maintenance being done on the detector and not all DCMs are in the run, this plot will show less than the 631 channels! Wait until a full-detector run is started and check this plot again.

Example plot of AutoStartDAQ not running and many missing channels:

This is BAD data



Good Subruns

This plot runs many data quality checks over the data and shows different data quality failures in different colors.

This is GOOD data

The plot should be white and with a flat rate.

The recent hours will show up as gray as we wait for reconstruction to be run over the data. As the reconstruction takes a while to run, a preliminary state of good or bad is shown based on low level quantities which shows up in a lighter shade of the same colour.
NOTE: GOOD data could be reclassified as bad with the extra reconstruction information but BAD data will never be reclassified as GOOD.

IMPORTANT: This plot is made using the beam data and therefore will be blank when there is no beam and will show a lower rate when the beam intensity is lower.
If we are running in a abnormal beam configuration, i.e horn off , off-target or low intensity running, these plots will reflect that and show the data to be bad.

Failure modes are:

Red: Failed Timing Peak
  • Look at the nearline timing peak plot and the OnMon timing plots. Click on the TQPlots folder and look at the TPlotALL and TPlotZoom of the Near Detector timing peak. White/red lines incicate where the timing peak should be located. If the timing peak is not visible (and there is beam) or is shifted compared to the white/red lines, call your DAQ expert.
  • If the timing peak is visible, send an email to with the time during which the plot has this failure mode.

GREEN: Failed DiBlock
Some part of the detector is either missing or has the wrong rate. Are channels also missing in the ‘Number of Active FEBs per Subrun’ plot? If so follow the instructions under that section.

If not follow the tests below in ORDER!

  1. Look at the detector configuration plot http://nusoft.fnal.gov/nova/datacheck/nearline/plots/FarDet-t02-P1GoodDataSelDetConfigDay.png to determine what region of the detector the problem is in.
  2. Are any DMCs warm? Look on the CSS GUI (APD temperature monitor on DAQ-CR-02) overview page. Are any of the DCMs (boxes) dark green instead of light green? For all normal running all of the detector should be cooled (LIGHT green). If they are not cooled either cool that DCM using the ‘Configure cold APD’ button or call an APD cooling expert straight away.
  3. Look at OnMon FEBHitRate and FEBHitRateMap plots (in the shifter folder). Do any DCMs have a high, low or zero hit rate? Do any DCMs in the event display not have hits in them? If so try issuing a “SYNC” (red button) from the TDU Control Interface. If this has fixed the issue, you see the effect pretty much straight away in OnMon and after ~10 mins in the Nearline.
  4. Look at OnMon FEBHitRateMapMipADC plot (in the RatePlot folder). Do any DCMs have a high, low or zero hit rate? If so, a DCM might be running at the wrong gain. Call Leon Mualem.
  5. If none of the above are true, call your DAQ expert.

LIGHT PURPLE: Failed Empty Spill

This means more than 5% for RHC (3% for FHC) of all beam spills have no events in them. This could be okay if we are running with a very low intensity beam. Check the POT beam plot in the nearline for what the intensity is.
Often this is seen for just a subrun when beam is only up for part of the hour or the beam intensity drops for some period and then it is okay.
Check the nearline beam quality plots to see if this is the case. If so, make a comment in the ECL. No other action is needed.

If we are not running at low intensity and beam is up, this means we are seeing less events that we should. This could be an issue with the thresholds or gains. call your DAQ expert.

BLUE: Hit Rate
This implies that the median hit rate in the MIP region was too high/low in the detector.

If can be caused if we are running in a odd beam configuration like without a horn or target. Check the nearline beam quality plots to see if we are. If we are, make a comment in the nearline form. No other action is needed.

Are any DCMs out of sync?
  • Look at OnMon FEBHitRate and FEBHitRateMap plots (in the shifter folder). Do any DCMs have a high, low or zero hit rate? Or do you see any DCMs on the event display that are not getting hits? If so try issuing a “SYNC” from the TDUControlInterface. If this has fixed the issue, you see the effect pretty much straight away in OnMon/Event display and after ~10 mins in the Nearline.
This could be an indication that the trigger rates are off.
  • Check if the Average Trigger and Spill Rates look ok, if they do not, call your DAQ expert.
  • Check the Trigger scalers on DAQ-CR-01. Are any trigger rates in alarm?
  • We expect the NuMI trigger = 0.7Hz, 1 Hz accelerator trigger = 1Hz, Cosmics = 1Hz, DDT Activity ~30-40Hz. If they are much higher or lower, call your DAQ expert.
This could be an indication that we are running at the wrong gain.
  • Look at OnMon FEBHitRateMapMipADC plot (in the RatePlot ) folder. Do any DCMs have a high, low or zero hit rate? If so a DCM might be running at the wrong gain. Call Leon Mualem.

PURPLE: NuMI Live Time
Subruns turn purple if the entire subrun had less than 1000 NuMI triggers worth of live time.

If there are purple subruns, check the following in order:

  1. If you see a subrun marked purple, check if beam is up. If beam is not up we may or may not be receiving the NuMI trigger. This is OK. Note in the logbook the times during which beam are down.
  2. If beam is up first check to see if there is good beam during these subruns. You can check this by looking at the nGoodBeamperSR plot. This is found in the Beam section of the Nearline plots, under the Near Detector heading. If this plot is showing zeroes for the purple subruns, we are receiving bad beam. This is not something we can fix on our end, just make an entry in the logbook.
  3. If there was good beam, check if a supernova trigger fired during these subruns. Look to see if the supernova trigger scalars have increased since the beginning of your shift (look back at screenshots in the ECL to compare). A supernova trigger can also cause >1 subrun/hour, leading to a bump in this plot. It is normal to see this failure mode during a supernova trigger, just make an entry in the logbook.
  4. If you do not see any subruns with bad beam or a supernova trigger, call a Data Quality expert. If none of them answer, proceed to calling your DAQ expert.

You may also see this for isolated subruns at the start or end of an run. If you see this for longer periods, call your DAQ expert and email as it may also be an issue with file processing.

ORANGE: Reconstruction/Slice rate
There were too many 2D tracks and/or there were too many/few slices per trigger.

Are any DCMs out of sync?
  • Look at OnMon FEBHitRate and FEBHitRateMap plots (in the shifter folder). Do any DCMs have a high, low or zero hit rate? Or do you see any DCMs on the event display that are not getting hits or do you see many short tracks ending at DCM boundaries? If so, try issuing a “SYNC” from the TDUControlInterface. If this has fixed the issue, you see the effect pretty much straight away in OnMon/Event display and after ~10 mins in the Nearline.
This could be indication that the trigger rates are off.
  • Check if the Average Trigger and Spill Rates look ok, if they do not, call your DAQ expert.
  • Check the Trigger scalers on DAQ-CR-01. Are any trigger rates in alarm?
  • We expect the cosmic trigger = 1Hz, 1 Hz accelerator trigger = 1Hz, NuMI = 0.7Hz. If they are much higher or lower call your DAQ expert.

This is POT weighted so if we are running without a horn or off target then expect this to be in alarm.

This could also be an issue with file processing email .

Black: Other
There were bad timestamps in the subrun.

This is an issue with file processing or corrupt files email .



OnMon FEB Hit Rate Spectrum vs. Time

This plot shows the rate of hits in the detector per subrun. The rate should remain constant unless the detector configuration or the beam intensity changes.

This is GOOD BEAM ON data

For the last and first subrun in the run low statistics could result in drops for that one subrun.

Sudden drops or increases in rate which do not coincide with a change in beam conditions indicate an issue.

  1. Are channels missing from the detector? Check the Number of Active FEBs per Subrun plot and follow the instructions there.
  2. Is the detector out of sync? In the Event Display can you see many short tracks ending on DCM boundaries? Is there a DCM with no hits? Look at OnMon FEBHitRate and FEBHitRateMap plots (in the shifter folder). Do any DCMs have a high, low or zero hit rate? If any of the above try issuing a “SYNC” from the TDUControlInterface. If this has fixed the issue so you see the effect pretty much straight away in OnMon, remember ND nearline will take ~1.5 hrs to update.
  3. Are any DCMs warm? Look on the CSS GUI (APD temperature monitor) overview page. Are any of the DCMs (boxes) dark green instead of light green? For all normally running all of the detector should be cool (LIGHT green). If they are not, cool the detector (using ‘configure cold APDs’ button) or call an APD cooling expert straight away.
  4. Look at OnMon FEBHitRateMapMipADC plot (in the RatePlot ) folder. Do any DCMs have a high, low or zero hit rate? If so a DCM might be running at the wrong gain. Call Leon Mualem.

If all the above does not fix the the issue as seen in Event Display or OnMon mins call your DAQ expert. Remember ND nearline will take ~1.5 hrs to update.

This is an example of BAD data.

This is BAD data.



Timing Peak

This plot shows the location of the timing peak over time.

For good BEAM ON data it should look like.
This is GOOD data

This plot is made using NuMI triggered files. It will be empty if there is no beam.

If the timing peak has shifted, call your DAQ expert straight away.

Look at the nearline timing peak plot and the OnMon timing plots. In OnMon click on the TQPlots folder and look at the TPlotALL and TPlotZoom of the Near Detector timing peak. White/red lines incicate where the timing peak should be located. If the timing peak is not visible (and there is beam) or is shifted compared to the white/red lines call your DAQ expert.