04142014 Minutes
Attending: Mat, Michael, Hayes, Eric F

Farm-16 - Power cycled. Some discussion about recovering HWWatch history. Michael thinks he has a way.
Discussion..... about new nearline processing design..... Michael will send an email around with details.

NDOS timing peak - Is visible.

Notes from forms
04/13/2014 21:03:46 P1
When we first started run 14692 (Partition 1), it was missing a DCM. This run was stopped very quickly, and 14693 had all 48 DCMs. This caused a couple of points to be 'not normal,' but they are understood and otherwise not interesting.
04/13/2014 13:21:20 P1
Please see 37483 about the worries I have about the gaps in plots. I believe somehow nearline got behind in processing (although I don't understand why -- it had a DSO scan during which to catch up) and after that we had RC server disconnects which make longer-than-usual subruns, which might be harder to process. I am hoping that explains that gap between 8ish and 10:45 (after that gaps are fine) -- but maybe something else went wrong that we don't know about!!

This is early in a run, so I expect more FEBs to fall out as it continues. For now, we are missing:

01:11:36 (always leaves run)
02:08:61 (always leaves run)
04:07:15 (bad module)

We are missing 18 pixels, all on 04:06:63, which is a terrible APD that is always half masked out and the other half is super hot. We also don't get calibration constants for this APD.

NuMI shows that we didn't take triggers overnight. This is known and documented in log book.
04/13/2014 12:42:48 P2
Please see my entry 37480 which lists the missing FEBs/pixels. It is the usual list of suspects. Over the life of this run, I expect a few more FEBs to drop out.

NuMI Trigger rate shows the loss of beam triggers last night.
04/12/2014 21:55:15
In NHits/Trigger plot, there is a jump at around 9 AM. See attached plot.
04/12/2014 11:59:49 P2
FEB Hit Rate spectra: yesterday there was a gap from Tian updating cooling and then a DSO scan. The gap from around 10 am this morning was Jan and I taking a DSO scan -- we did it in record time of less than 45 minutes!!!

Instability in number of DCMs - dcm-9-3 was out for most of the night. This was solved by issuing a sync. This accounts for many strange features in the plots, including the spike of empty events.

Partition 2 reliably has the same FEBs falling out of runs. We just took a DSO scan AGAIN, with the same results as I have been tracking all week. The FEBs always missing from the run are:

05:07:59 (bad module)
10:04:33 (this has had problems since installation - Jan removed it from the mask by hand)

Watching missing pixels, these have also been stable all week and despite multiple DSO scans. There are 25 missing pixels, located at:

05:01:22 - 6 pixels
05:03:27 - 16 pixels
04/12/2014 10:22:28 P1
See Jan's forthcoming entry about average number of hits per spill. His is also going to make an entry about empty events.
04/12/2014 02:27:16
The gaps possibly due to DSO scan
04/11/2014 23:07:57 P1
Data taking was down from ~15:00-16:30, and came back up after DSO scan finished. This is the only real gap in data.
We have lost 5 FEBs since the last run started, we have been stable at 3066 for about an hour.
Drops in pixels per subrun correspond to drops in FEBs.

There's a steady decrease (5%) in the average number of hits per spill during the course of running.
Trigger rate shows many outliers below the average. Beam has been steady during running. Actually these same drops are observed in the cosmic rate as well, so it's probably OnMon related.
04/11/2014 15:20:15
This partition has a lot of masked FEBs which should come back after the next DSO scan.