Project

General

Profile

NDOS CommissionMtng 20120521

May 21 Commisioning Meeting: Jon, Martin, Athans, Evan, Gavin, Peter Lucas, Peter S, Leon, Zukai, Mat, Chad, Gary, Xueying

1.  Detector Update
Booster and NUMI streams are off

Last week installed more APDs: 10 Silicone, 8 parylene (7 x3, 1x),

Old FEB enable configuration was being used as "latest" in RC. Workaround is to manually select newest config with the "more" option.

Leon - Found the run dead last night at around 8. Random APD going into alarm. Seemed to progress as the evening progressed.

Leon unplugged a FEB while running and everything else ran normally. DCM died when he plugged back it.

DSO scans are producing intermittent data? We will monitor that with scans this week.

Second timing chain being installed NOW.

Muon catcher APD and FEBs have been removed from the muon catcher. We will make sure it's clear to those starting runs to NOT bring those into the DAQ.

Leon pointed out a RC glitch with remembering BNEVB configurations.

2.  DataCheck Update- Jan Emailed in the following report

Monday: First run of the day after DSO scan was 13822, from 10:12am to 5:54pm, and subrun 0 seemed to have some real problems in terms of channels dropping out. We started out with 5239 raw active channels and ended the run with 4697 raw active channels. Here is the breakdown by subrun and febs on dcms:
  run  | subrun | ndcms | dcm11 | dcm12 | dcm13 | dcm21 | dcm22 | dcm23 | dcm31 | dcm32 | dcm33 | dcm41 | dcm42 
-------+--------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------
 13822 |      0 |    11 |    24 |    29 |    14 |    23 |    24 |    13 |     4 |    13 |     3 |    12 |    19
 13822 |      1 |    10 |    24 |    29 |    14 |    23 |    24 |     8 |     4 |    11 |     0 |    12 |    17
 13822 |      2 |    10 |    24 |    29 |    14 |    23 |    24 |     8 |     4 |    11 |     0 |     8 |    17
 13822 |      3 |    10 |    24 |    29 |    14 |    23 |    24 |     8 |     4 |    11 |     0 |     8 |    17
 13822 |      4 |    10 |    24 |    29 |    14 |    23 |    22 |     8 |     4 |    11 |     0 |     8 |    17
 13822 |      5 |    10 |    24 |    29 |    14 |    23 |    22 |     8 |     4 |    11 |     0 |     8 |    17
 13822 |      6 |    10 |    24 |    29 |    14 |    23 |    22 |     8 |     4 |    11 |     0 |     8 |    17
 13822 |      7 |    10 |    24 |    29 |    14 |    23 |    22 |     8 |     4 |    11 |     0 |     8 |    17

Another peculiarity was that during subrun 4 we recorded 106 empty events. I'm guessing this was in conjunction with two FEBs disappearing from DCM2-3.

The overnight run, 13823, seemed a lot smoother. We lost one FEB in subrun 3 from DCM2-1, but then all was quiet until subrun 15. This corresponds to between 9:30am and 10:30am, and we lost 5 FEBs. Then another 2 during subrun 16.
  run  | subrun | ndcms | dcm11 | dcm12 | dcm13 | dcm21 | dcm22 | dcm23 | dcm31 | dcm32 | dcm33 | dcm41 | dcm42 
-------+--------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------
 13823 |     15 |    11 |    24 |    29 |    14 |    22 |    24 |    13 |     4 |    13 |     3 |    11 |    19
 13823 |     16 |    11 |    22 |    29 |    14 |    20 |    24 |    13 |     4 |    13 |     3 |    10 |    19
 13823 |     17 |    11 |    22 |    29 |    14 |    20 |    24 |    13 |     4 |    12 |     2 |    10 |    19

There have also been multiple start run forms where it's looking like the first one is an automatic entry and then the second one by the shifter. It seems the appropriate information is included in the form generated by the novadaq user and conflicting information from the shifter is quite confusing.

Tuesday:
The two next runs 13824 and 13825 were short ones, corresponding to DCMs being taken out of the line-up to facilitate APD work. The overnight run, 13826, was really stable without DCM4-2 and lasted until Wednesday at 9am.

Wednesday:
Run 13827 was a rough one. During subrun 0 we lost 21 FEBs and 11 FEBs during subrun1
run  | subrun | ndcms | dcm11 | dcm12 | dcm13 | dcm21 | dcm22 | dcm23 | dcm31 | dcm32 | dcm33 | dcm41 | dcm42 
-------+--------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------
 13827 |      0 |     8 |    24 |    29 |    14 |    23 |    24 |    13 |     4 |     0 |     0 |    11 |     0
 13827 |      1 |     8 |    24 |    29 |    14 |    23 |    10 |     6 |     4 |     0 |     0 |    11 |     0
 13827 |      2 |     8 |    24 |    23 |    12 |    23 |    10 |     4 |     3 |     0 |     0 |    11 |     0

The run lasted for 5 subruns, with 219 empty events at the end of subrun 4 and 476 empty events at the beginning of subrun 5. This is very likely due to work on sections of the detector that were not being read out, affecting sections of the detector that were in the read out. 

Run 13828 had enough missing DCMs to drive the number of noise only events up, these are events where we do not have enough correlated hits to form a physics slice. 5500 noise events out of 35,600 compared to 4000 in a previous run with a comparable number of raw active channels, but more spread out.

The overnight run, 13829, was relatively stable after a bumpy start, lost 743 channels right of the bat, if you trust the automatically generated number of raw active channels from novadaq of 3910, but only 3167 got hit.

Thursday:
Run 13830 was pretty terrible, as I think we have come to expect when there is work happening at the NDOS surface building. Here is a list of the number of FEBs per DCM per subrun:
  run  | subrun | ndcms | dcm11 | dcm12 | dcm13 | dcm21 | dcm22 | dcm23 | dcm31 | dcm32 | dcm33 | dcm41 | dcm42 
-------+--------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------
 13830 |      0 |     6 |    24 |    29 |    14 |    23 |    24 |     0 |     4 |     0 |     0 |     0 |     0
 13830 |      1 |     6 |    23 |    29 |    14 |    23 |    24 |     0 |     3 |     0 |     0 |     0 |     0
 13830 |      2 |     6 |    23 |    29 |    14 |    23 |    24 |     0 |     3 |     0 |     0 |     0 |     0
 13830 |      3 |     5 |    23 |    28 |    14 |    23 |    24 |     0 |     0 |     0 |     0 |     0 |     0
 13830 |      4 |     5 |    23 |    27 |    13 |    23 |    24 |     0 |     0 |     0 |     0 |     0 |     0
 13830 |      5 |     5 |    23 |    27 |    13 |    23 |    19 |     0 |     0 |     0 |     0 |     0 |     0
 13830 |      6 |     5 |    23 |    25 |    12 |    23 |    13 |     0 |     0 |     0 |     0 |     0 |     0
 13830 |      7 |     5 |    23 |    22 |     7 |    23 |    10 |     0 |     0 |     0 |     0 |     0 |     0
 13830 |      8 |     5 |    23 |    22 |     7 |    23 |    10 |     0 |     0 |     0 |     0 |     0 |     0
 13830 |      9 |     5 |    23 |    22 |     7 |    23 |    10 |     0 |     0 |     0 |     0 |     0 |     0

what might be worse is that we recorded 7,652 21,617 and 6,933 empty events in subruns 0, 1, and 2 respectively. 
Attempt at getting an overnight run started, failed.

Friday:
After some attemps got going with run 13833 and the Corrupt Data Messages Evan noted in the log book, did not translate into empty events. We did pick up some empty events, 1148, at the beginning of the last subrun, 7. On a side note, I could put a better timestamp on when that happened, but these numbers are coming from the nearline table, not the usual datacheck files. We've run into an issue where datafiles have not been reprocessed since 2012-05-06. Jarek is looking into this.
 run  | subrun | ndcms | dcm11 | dcm12 | dcm13 | dcm21 | dcm22 | dcm23 | dcm31 | dcm32 | dcm33 | dcm41 | dcm42 
-------+--------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------+-------
 13833 |      0 |     6 |    24 |    29 |    14 |    23 |    24 |     0 |     4 |     0 |     0 |     0 |     0
 13833 |      1 |     6 |    24 |    29 |    14 |    22 |    24 |     0 |     4 |     0 |     0 |     0 |     0
 13833 |      2 |     6 |    24 |    29 |    14 |    22 |    24 |     0 |     4 |     0 |     0 |     0 |     0
 13833 |      3 |     6 |    24 |    25 |     9 |    22 |    21 |     0 |     3 |     0 |     0 |     0 |     0

The other subruns don't show a change in the number of active FEBs per DCM.
Overnight run, 13834, started with 4 empty events, the rest of it ran quite smooth, only lost 2 FEBs in 18 hours of running.

Saturday and Sunday:
Run 13835 ran great, with nobody touching the detector, and rolled over into run 13836 as of 40 minutes ago. The run was stopped 20 minutes in order to take a DSO scan. This is where the log book entries stop so that's where this report will end. 

The short version:

When we do work on the detector, data quality is poor, whether the area of interest is included in the data stream or not. This looks like mostly FEB buffer overflows, but empty events creep up from time to time. Night and weekend runs are stable. Jarek is looking into the fact that batch reprocessing seems to be faulty at the moment. He is also working on reworking the fits to determine when we encounter bad periods for the nearline plots, which should improve the efficiency of the emails we should be receiving when something isn't looking right.

3.  Installation Plans for the week
Collect stable running with new APDs. Cool some of the side APDs?
Install more APDs beginning on Wednesday top of diblock 3 (block 5). Silicone APD.

4.  DAQ Update and Plans for the week
RC DB interface issue causing loading of bad configurations. (Kurt was working on this as of this morning.)

Partitioning

New FEB firmware was released. Fixes HV inversion issue. Needs coordination with DCS.

Denis using 2-1 for TEC control

5.  DCS Update and Plans for the week (Athans)

Shifters should exercise new Cooling GUI.

6.  AOB
Xueying Huyan and Evan Niner are available to help with shift work and installation as needed