Project

General

Profile

Bug #24253

Nova Test Beam Scalers logged on Event 37 missing 2/3 of data

Added by Dennis Nicklaus 3 months ago. Updated 3 months ago.

Status:
Assigned
Priority:
Normal
Category:
MOOC Framework
Target version:
-
Start date:
03/31/2020
Due date:
% Done:

0%

Estimated time:
Duration:

Description

For devices read from the NOVASC front-end, logging them on event 37 only captures about 1/3 of the data.
Devices include Z:NOVTSC01 - Z:NOVTSC31. They are logged on event 37 in the NOvAN Event list. But they only show up in the logger about 1/3 of the time -- or 2/3 of the 37s are missed somehow. NovaSC is a standard mooc front-end in the Meson test beam area. It only has multicast clock. Shown is data from the data loggers including a different device from a different frontend that is logged on the same event, as well as Event37 device totals. Event 37 usually happens about once per minute and is the spill of data to the test beam area. The NovaSC devices are pretty obviously only logging about 1/3 as much as they should.


    Lumberjack Read Data List          Tue   31-MAR-20 11:17:11

    Engineering units
    X = TIME  Lumberjack format

    T1 = 26-FEB-2020 10:19:12.000    T2 = 26-FEB-2020 12:19:12.000

     Trace 1                 Trace 2                 Trace 3
           48 points                       15 points                       47 points
     F:MC6IC ppp    FixTr2             Z:NOVTSC01 cnt    NOvAN              G:E37TOT cnt    EventC

        Time            Y1              Time            Y2              Time            Y3
Wed Feb 26 11:31:50.832    0               Wed Feb 26 11:33:50.969    6804            Wed Feb 26 11:32:38.992    1873134     
Wed Feb 26 11:32:50.907    0               Wed Feb 26 11:36:51.175    52186           Wed Feb 26 11:33:39.052    1873135     
Wed Feb 26 11:33:50.966    79900000        Wed Feb 26 11:40:51.377    65639           Wed Feb 26 11:34:39.115    1873136     
Wed Feb 26 11:34:51.034    193640000       Wed Feb 26 11:43:51.572    60558           Wed Feb 26 11:35:39.186    1873137     
Wed Feb 26 11:35:51.118    283410000       Wed Feb 26 11:46:51.747    58179           Wed Feb 26 11:36:39.259    1873138     
Wed Feb 26 11:36:51.190    829550000       Wed Feb 26 11:50:51.992    67797           Wed Feb 26 11:37:39.305    1873139     
Wed Feb 26 11:37:51.236    879840000       Wed Feb 26 11:53:52.174    58993           Wed Feb 26 11:38:39.354    1873140     
Wed Feb 26 11:38:51.285    1075830000      Wed Feb 26 11:56:52.321    60403           Wed Feb 26 11:39:39.402    1873141     
Wed Feb 26 11:39:51.332    1035880000      Wed Feb 26 11:59:52.501    63840           Wed Feb 26 11:40:39.460    1873142     
Wed Feb 26 11:40:51.391    1065490000      Wed Feb 26 12:00:52.555    59422           Wed Feb 26 11:41:39.531    1873143     
Wed Feb 26 11:41:51.465    842710000       Wed Feb 26 12:03:52.727    59206           Wed Feb 26 11:42:39.608    1873144     
Wed Feb 26 11:42:51.539    1073950000      Wed Feb 26 12:06:52.931    51334           Wed Feb 26 11:43:39.655    1873145     
Wed Feb 26 11:43:51.586    988880000       Wed Feb 26 12:09:53.143    65111           Wed Feb 26 11:44:39.718    1873146     
Wed Feb 26 11:44:51.649    980890000       Wed Feb 26 12:13:53.382    68253           Wed Feb 26 11:45:39.775    1873147     
Wed Feb 26 11:45:51.705    1053270000      Wed Feb 26 12:16:53.564    65527           Wed Feb 26 11:46:39.830    1873148     
Wed Feb 26 11:46:51.760    979010000                                       Wed Feb 26 11:47:39.898    1873149     
Wed Feb 26 11:47:51.829    1095570000                                      Wed Feb 26 11:48:39.962    1873150     
Wed Feb 26 11:48:51.893    1013320000                                      Wed Feb 26 11:49:40.024    1873151     
Wed Feb 26 11:49:51.954    1086170000                                      Wed Feb 26 11:50:40.075    1873152     
Wed Feb 26 11:50:52.005    1050920000                                      Wed Feb 26 11:51:40.137    1873153     
Wed Feb 26 11:51:52.054    1064550000                                      Wed Feb 26 11:52:40.202    1873154     
Wed Feb 26 11:52:52.118    1070660000                                      Wed Feb 26 11:53:40.256    1873155     
Wed Feb 26 11:53:52.173    1026950000                                      Wed Feb 26 11:54:40.312    1873156     
Wed Feb 26 11:54:52.228    962090000                                       Wed Feb 26 11:55:40.361    1873157     
Wed Feb 26 11:55:52.276    940470000                                       Wed Feb 26 11:56:40.405    1873158     
Wed Feb 26 11:56:52.320    1045750000                                      Wed Feb 26 11:57:40.465    1873159     
Wed Feb 26 11:57:52.382    970080000                                       Wed Feb 26 11:58:40.523    1873160     
Wed Feb 26 11:58:52.438    1078180000                                      Wed Feb 26 11:59:40.584    1873161     
Wed Feb 26 11:59:52.499    1097920000                                      Wed Feb 26 12:00:40.637    1873162     
Wed Feb 26 12:00:52.553    996870000                                       Wed Feb 26 12:01:40.700    1873163     
Wed Feb 26 12:01:52.615    1065490000                                      Wed Feb 26 12:02:40.753    1873164     
Wed Feb 26 12:02:52.667    971960000                                       Wed Feb 26 12:03:40.809    1873165     
Wed Feb 26 12:03:52.725    1070190000                                      Wed Feb 26 12:04:40.885    1873166     
Wed Feb 26 12:04:52.800    1026950000                                      Wed Feb 26 12:05:40.956    1873167     
Wed Feb 26 12:05:52.872    1082410000                                      Wed Feb 26 12:06:41.014    1873168     
Wed Feb 26 12:06:52.928    915090000                                       Wed Feb 26 12:07:41.083    1873169     
Wed Feb 26 12:07:52.997    1000160000                                      Wed Feb 26 12:08:41.155    1873170     
Wed Feb 26 12:08:53.072    1061260000                                      Wed Feb 26 12:09:41.225    1873171     
Wed Feb 26 12:09:53.142    1031650000                                      Wed Feb 26 12:10:41.285    1873172     
Wed Feb 26 12:10:53.202    997340000                                       Wed Feb 26 12:11:41.344    1873173     
Wed Feb 26 12:11:53.260    974780000                                       Wed Feb 26 12:12:41.400    1873174     
Wed Feb 26 12:12:53.317    962560000                                       Wed Feb 26 12:13:41.466    1873175     
Wed Feb 26 12:13:53.381    1110610000                                      Wed Feb 26 12:14:41.531    1873176     
Wed Feb 26 12:14:53.448    1026010000                                      Wed Feb 26 12:15:41.592    1873177     
Wed Feb 26 12:15:53.507    1030710000                                      Wed Feb 26 12:16:41.646    1873178     
Wed Feb 26 12:16:53.562    1027420000                                      Wed Feb 26 12:17:41.704    1873179     
Wed Feb 26 12:17:53.619    1032120000                                      Wed Feb 26 12:18:41.771    1873180     
Wed Feb 26 12:18:53.687    967730000                                                                   

History

#1 Updated by Dennis Nicklaus 3 months ago

It's not unique to the dataloggers, nor $37, nor those particular devices. So it looks like the multicast clock. NovaSC is not joining to any specific clock, just the default TCLK one.

I did a little experiment that shows it takes ACL roughly 3x longer than 50s to accumulate 10 Event02 readings from that frontend's guaranteed readable device. I printed the time just before entering ACL and immediately after the last reading was done. So there's a little variability in my response. But just a few seconds.

clx58$ date
Tue Mar 31 11:52:27 CDT 2020
clx58$ acl

ACL> loop 10;read/pendwait g:d0egrs/ftd=evt02;endloop

G:D0EGRS = -.389754 Test
G:D0EGRS = -.78050888 Test
G:D0EGRS = -.99552637 Test
G:D0EGRS = .99066806 Test
G:D0EGRS = -.91855133 Test
G:D0EGRS = .76528108 Test
G:D0EGRS = -.32153881 Test
G:D0EGRS = .64724761 Test
G:D0EGRS = .1819472 Test
G:D0EGRS = -.14164756 Test

ACL>
clx58$ date
Tue Mar 31 11:55:09 CDT 2020

#2 Updated by Richard Neswold 3 months ago

  • Status changed from New to Assigned

Comparing UCDA to MI4, for instance (ignore the first column since I can't easily run the commands simultaneously.)

UCDA:

Listing of HISTOGRAM of EVENTS: non-zero; current=3, SC=25 Secs.
            current cycle--------->oldest cycle
Histogram =    03     02     01     00 
event 00:       0;     1;     1;     1;
event 02:       5;    12;    12;    12;
event 07:    1635;  3928;  3927;  3927;
event 0b:       5;    12;    12;    12;
event 0c:     375;   900;   900;   900;
event 0f:     375;   900;   900;   900;
event 10:       5;    12;    12;    12;
event 11:     365;   876;   876;   876;
event 12:       5;    12;    12;    12;
event 17:       5;    12;    12;    12;
event 18:     375;   900;   900;   900;
event 1a:       5;    12;    12;    12;
event 1b:       5;    12;    12;    12;
event 1e:       5;    12;    12;    12;
event 1f:       5;    12;    12;    12;
event 22:       3;     6;     6;     6;
event 25:       3;     6;     6;     6;
event 26:       3;     6;     6;     6;
event 2a:       3;     6;     6;     6;
event 2c:       3;     6;     6;     6;
event 2f:       3;     6;     6;     6;
event 3c:       0;     1;     1;     1;
event 4c:     365;   876;   876;   876;
event 53:       5;    12;    12;    12;
event 72:       5;    12;    12;    12;
event 7a:      21;    42;    42;    42;
event 7b:       3;     6;     6;     6;
event 7d:       3;     6;     6;     6;
event 8f:      25;    60;    60;    60;
event a2:       1;     1;     1;     1;
event ac:       2;     6;     6;     6;
event b3:      38;    90;    90;    90;
event b4:      19;    45;    45;    45;
event ba:     500;  1200;  1200;  1200;
event e3:      19;    45;    45;    45;
event e6:      19;    45;    45;    45;
event ef:      19;    45;    45;    45;

MI4:

Listing of HISTOGRAM of EVENTS: non-zero; current=3, SC=27 Secs.
            current cycle--------->oldest cycle
Histogram =      3      2      1      0
event 00:        0      1      1      1
event 02:        5     12     12     12
event 07:    19789  43200  43200  43200
event 0b:        5     12     12     12
event 0c:      413    900    900    900
event 0f:      413    900    900    900
event 10:        5     12     12     12
event 11:      403    876    876    876
event 12:        5     12     12     12
event 17:        5     12     12     12
event 18:      412    900    900    900
event 1a:        5     12     12     12
event 1b:        5     12     12     12
event 1e:        5     12     12     12
event 1f:        5     12     12     12
event 22:        3      6      6      6
event 25:        3      6      6      6
event 26:        3      6      6      6
event 2a:        3      6      6      6
event 2c:        3      6      6      6
event 2f:        3      6      6      6
event 3c:        0      1      1      1
event 4c:      402    876    876    876
event 53:        5     12     12     12
event 72:        5     12     12     12
event 7a:       21     42     42     42
event 7b:        3      6      6      6
event 7d:        3      6      6      6
event 8f:       28     60     60     60
event a2:        1      1      1      1
event ac:        2      6      6      6
event b3:       41     90     90     90
event b4:       21     45     45     45
event ba:      550   1200   1200   1200
event e3:       21     45     45     45
event e6:       20     45     45     45
event ef:       20     45     45     45

I don't know why the VUCD library sees so few $07 events. Everything else is spot-on. No $37s fired, so I can't compare those numbers.

#3 Updated by Richard Neswold 3 months ago

I don't know why the VUCD library sees so few $07 events.

Although this might explain why the multicast doesn't report 48 $07s occurred in each frame.

#4 Updated by Dennis Nicklaus 3 months ago

Unfortunately, I can't login to novasc's shell right now, and I apparently just missed my chance for a reboot this week. So I can't run trig_event_listing there.
The closest I came is stcrfi, which is at Meson, at least. Novasc is back in a remote corner of its own.
Stcrfi (multicast, set to listen to TCLK multicast for the below measurement) shows the correct # of $02s at least.
From Stcrfi:


-> trig_event_listing
Listing of HISTOGRAM of EVENTS: non-zero; current=1, SC=10 Secs.
            current cycle--------->oldest cycle
Histogram =    01     00     03     02 
event 00:       0;     1;     1;     1;
event 02:       2;    12;    12;    12;
event 07:     160;   900;   900;   900;

#5 Updated by Dennis Nicklaus 3 months ago

Well that's too bad. After a VME reboot, the front-end is responding just fine.

ACL> loop 10;read/pendwait u:upt9d1/ftd=evt02; print currentDate;endloop

gives me all the data I expect, and trig_event_listing looks beautiful, too.
novasc->trig_event_listing
Listing of HISTOGRAM of EVENTS: non-zero; current=3, SC=34 Secs.
            current cycle--------->oldest cycle
Histogram =    03     02     01     00 
event 00:       0;     1;     1;     1;
event 02:       7;    12;    12;    12;
event 07:     510;   900;   900;   900;
event 0b:       6;    12;    12;    12;



Also available in: Atom PDF