Buffer Node Error Checklist

When you see a "bnevb process unresponsive" popup error window or if a buffer node turns yellow in DAQ Application Manager, do/check the following.

1) Start an ECL entry and log the buffer node that threw the error. You can find which buffer node is the offender by looking in DAQ application manager (click on the yellow box) or looking at the popup error window from run control.

2) Verify that the run still seems to be proceeding as normal. Contact the DAQ expert immediately if any of the below is not as it should be.

  • Is run control red?
  • Are the triggers in the Global Trigger GUI still advancing normally?
  • Are any buffer nodes yellow in DAQ Application Manager? If so, this is fine. Make a note of it in your ECL entry and watch it as your shift continues. It will likely stay yellow for ~3 hours.
  • Does the EVD look normal? Watch it for ~1 minute and look for the following (feel free to stop the auto advance and manually click forwards through events):
    -> No missing DCMs
    -> NuMI and PerCal triggers appear to have the full 550 usecs of readout (look for the full rainbow blue->red in the time histogram in the bottom left.) DDT triggers can be of variable length so it is perfectly normal for them not to have the full 550 usec of readout.
    -> Normal mixture of triggers, some DDT, some PerCal, maybe a few NuMI
    -> Tracks from cosmic rays look pretty much normal? Straight lines, not broken at DCM boundaries?
  • In OnMon, look at the plot titled “AEVsHour” in the “Shift” folder. Do you see a burst of the error labeled “Incomplete Events.” This can be normal in this situation, so if you see it, there is no need to contact an expert. BUT if this persists for more than 5 minutes (each bin is 1 minute wide, so 5 consecutive bins with this problem is bad) call the DAQ expert.

3) Add to the ECL entry that you started when this happened that you have verified all of the above is fine.

4) Dismiss the popup error box with the “buffer node error.”

5) Call the DAQ expert and have them check the incomplete event rate from the offending buffer node and check that this node is otherwise running normally.

6) Fill out a stable runs checklist and continue to watch the above things. SPECIFICALLY continue to watch the AEVsHour plot in OnMon until the buffer node turns green again in DAQ Application Manager.