Project

General

Profile

Support #25041

Timestamps between front-ends for a given event are not the same

Added by Beau Harrison about 2 months ago. Updated about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Start date:
10/01/2020
Due date:
% Done:

0%

Estimated time:
Duration:

Description

This is copied from a GitHub repo, https://github.com/fermi-controls/timestamp-synchrony/blob/main/results/README.md

The assumptions here are that when a request is made across multiple front-ends on the same event, in this case $02, that the timestamps be the same. This includes being within the same ms.

Curated Results

These are a collection of observations from the [raw output](./rawOutput.txt).

Future Efforts

Dennis Nicklaus recommended running the test again and comparing the results for consistency.

Unresponsive Nodes

These nodes weren't marked as down but a request on event to it's guaranteed readable device didn't respond.

Node Guaranteed Readable
---- -------------------
AILAB Z:PDSINE
BBLOND U:TSTADC
DBPM10 U:TSTD96
DBPM20 U:TSTD97
DBPM30 U:TSTD98
DBPM40 U:TSTD99
DBPM50 U:TSTD9A
DBPM60 U:TSTD9B
IDAMP I:DMPGUA
IRM562 Z:S562SR
IRM571 G:S571SR
IRM586 E:S586SR
IRM588 E:S588SR
LIP628 L:S628SR
LIP728 L:S728SR
MIMISC Z:MIMISC
MIWIRS Z:MIWIRS
N60TOR E:TOR101
NMLBBM Z:NMLBBM
NMLTR2 U:TSTD9C
NUMIHN U:VIDBB4
QXRTEV S:NQXFSM
RCYSCH Z:RCYSCH
RRXDMP R:UPTBEB
SBPMF3 G:0C05V0
TBLMF1 U:VIDC7C
TBLMF2 U:VIDC7D
TBLMF3 U:VIDC7E
TBLMF4 U:VIDC7F
TESTZY Z:TESTZY
UBLM10 U:VIDC67
UBLM30 U:VIDC69
UBLM3A U:VIDC7A
UBLM3B U:VIDC68
UBLM50 U:VIDC6B
UBLMM4 U:VIDC66

Duplicate Guaranteed Readable Device

There is evidence of duplicate guaranteed readable devices being use. This shouldn't be the case.

Node Guaranteed Readable
---- -------------------
BOOSTR B:SLD1
BOOST1 B:SLD1

`BOOST1` is the new name of `BOOSTR` so `BOOSTR` is an alias for `BOOST1`. This is a case where duplicate guaranteed readable devices is understood to be ok.

Timestamps

The modal timestamp was determined to be `1601496697594 UTC`.

Summary Count of Timestamps and Deltas

This is a list of deltas calculated from the modal delta and a count of how many instances of each delta were found. The idea is that common operating modes may influence how far from accurate a timestamp is. So, finding common modes amongst these deltas could point to a problem.

Timestamp Delta Count (▼) Timestamp (▼) Delta Count
------------- ----- --------- - -------------- ----- -----
1601496697594 0 428 538037594 -1600958660000 1
1601496697608 14 27 29490775594 -1572005922000 1
1601496697595 1 17 1600514811608 -981885986 3
1601496696794 -800 11 1600859632774 -637064820 1
1601496697600 6 7 1600859632799 -637064795 1
1601496697699 105 3 1601496624594 -73000 1
1601496697607 13 3 1601496671595 -25999 1
1601496697593 -1 3 1601496682594 -15000 1
1601496696793 -801 3 1601496685594 -12000 1
1600514811608 -981885986 3 1601496692600 -4994 1
1601496696727 -867 2 1601496692973 -4621 1
1601498638862 1941268 1 1601496692977 -4617 1
1601496829594 132000 1 1601496692997 -4597 1
1601496697649 55 1 1601496692999 -4595 1
1601496697629 35 1 1601496693014 -4580 1
1601496697596 2 1 1601496693015 -4579 1
1601496697578 -16 1 1601496693246 -4348 1
1601496694462 -3132 1 1601496693269 -4325 1
1601496693324 -4270 1 1601496693281 -4313 1
1601496693309 -4285 1 1601496693282 -4312 1
1601496693289 -4305 1 1601496693283 -4311 1
1601496693288 -4306 1 1601496693288 -4306 1
1601496693283 -4311 1 1601496693289 -4305 1
1601496693282 -4312 1 1601496693309 -4285 1
1601496693281 -4313 1 1601496693324 -4270 1
1601496693269 -4325 1 1601496694462 -3132 1
1601496693246 -4348 1 1601496696727 -867 2
1601496693015 -4579 1 1601496696793 -801 3
1601496693014 -4580 1 1601496696794 -800 11
1601496692999 -4595 1 1601496697578 -16 1
1601496692997 -4597 1 1601496697593 -1 3
1601496692977 -4617 1 1601496697594 0 428
1601496692973 -4621 1 1601496697595 1 17
1601496692600 -4994 1 1601496697596 2 1
1601496685594 -12000 1 1601496697600 6 7
1601496682594 -15000 1 1601496697607 13 3
1601496671595 -25999 1 1601496697608 14 27
1601496624594 -73000 1 1601496697629 35 1
1600859632799 -637064795 1 1601496697649 55 1
1600859632774 -637064820 1 1601496697699 105 3
29490775594 -1572005922000 1 1601496829594 132000 1
538037594 -1600958660000 1 1601498638862 1941268 1

Nodes Grouped By Delta

The idea here is that nodes with a common implementation may have a similar timestamp offset.

Delta Nodes
----- -----
1 CMLLRF, E906MW, HTSK13, HTSK39, HWRRFI, NMGUNI, NMGWIL, NMLBP2, NMLBP3, NMLGP1, NMLHOM, NMLLD1, NMLRFI, P2ITB1, STCRFI, UCDHIN, XTBPM1
2 HINSMC
6 BADMAB, MILLRF, MRF, NBPB65, NBPM6N, NBPM6S, NBPMOV
13 CMTIL1, CMTIL2, CMTIL3
14 AP0TOR, BOOST1, BOOST2, BOOSTR, MCRBCC, MESON, MI1, MI2, MI3, MI4, MUONFE, MWF2P3, MWIRE0, MWIRE1, MWIRE2, MWIRE3, MWMB10, MWMB12, MWMS1, MWMTA, NEUTR, NUMI, QXRMI, SWICT, SWYD, TCLK, TEVA
35 NMLWPM
55 IRM593
105 MT1, PXMC1, PXMC2
132000 AP5001
1941268 MES653
-637064820 BOODAM
-637064795 BPM399
-4325 BPM400
-4580 CLX38
-1 EVENTH, EVENTN, LRHVAC
-16 MRFD
-800 IRM58E, IRM64E, NML655, NML656, NML657, NML658, NML659, NML65A, NML65B, NML65C, NML65D
-801 NML654, PXE690, PXE691
-867 CMT60A, CMT60B
-3132 LVTEST
-4285 NMLA
-4270 UCDNML
-4305 ??
-4306 OTISVW
-4311 SSRRFI
-4312 XMLRPC
-4313 NMLTOR
-4348 IOTASR
-4579 SSRRFJ
-4595 MWDIPM
-4597 MTACH
-4617 VMEJEF
-4621 RRLLD
-4994 TLGTST
-12000 PRBOPR
-15000 TLG
-25999 NMLTRG
-73000 RRLLNV
-981885986 MWMI14, NMAAT, NUMI65
-1572005922000 SRFLPR
-1600958660000 BLLRFD

Keepers Grouped By Delta

The idea here is that particular delta may be associate with the node keeper.

Delta Keepers
----- -------
1941268 Mike Kucera
132000 Jerry Firebaugh
105 Rich Neswold, Rich Neswold, Rich Neswold
55 Mike Kucera
35 Nathan Eddy
14 John Diamond, Rich Neswold, Rich Neswold, Rich Neswold, Rich Neswold, Rich Neswold, Rich Neswold, Rich Neswold, Rich Neswold, Rich Neswold, Rich Neswold, Jianming You, Jianming You, Jianming You, Jianming You, Jianming You, Jianming You, Jianming You, Jianming You, Jianming You, Rich Neswold, Rich Neswold, Jerry Firebaugh, Jianming You, Rich Neswold, Rich Neswold, Rich Neswold
13 Dennis Nicklaus, Dennis Nicklaus, Dennis Nicklaus
6 Kevin Martin, Philip Varghese, Ed Cullerton, John Diamond, John Diamond, John Diamond, John Diamond
2 Rich Neswold
1 Philip Varghese, Jianming You, Dennis Nicklaus, Dennis Nicklaus, Dennis Nicklaus, Dennis Nicklaus, Dennis Nicklaus, David Slimmer, David Slimmer, Jerry Firebaugh, Dennis Nicklaus, Rich Neswold, Dennis Nicklaus, John Diamond, Dennis Nicklaus, Rich Neswold, John Diamond
-1 Charlie King, Charlie King, Jerry Firebaugh
-16 Ed Cullerton
-800 Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera, Mike Kucera
-801 Mike Kucera, Mike Kucera, Mike Kucera
-867 Mike Kucera, Mike Kucera
-3132 Jianming You
-4270 Rich Neswold
-4285 Rich Neswold
-4306 Randy Thurman-Keup
-4305 Jianming You
-4311 Dennis Nicklaus
-4312 Charlie King
-4313 John Diamond
-4325 Jianming You
-4348 Randy Thurman-Keup
-4579 Dennis Nicklaus
-4580 Denise Finstrom
-4595 Jianming You
-4597 Jianming You
-4617 Nathan Eddy
-4621 Brian Chase
-4994 Linden Carmichael
-12000 Ed Cullerton
-15000 Linden Carmichael
-25999 John Diamond
-73000 Ed Cullerton
-637064795 Jianming You
-637064820 Dennis Nicklaus
-981885986 Jianming You, Jianming You, Jianming You
-1572005922000 Rich Neswold
-1600958660000 William Pellico

Software Grouped By Delta

The idea here is that particular delta may be associate with the node software. Preferably we know the timing driver and version for these nodes so we get a better idea of if there is an implementation error.

Delta Software
----- --------
1 MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, ACSys/FEF, MOOC, MOOC, MOOC
-1 Java, Java, Java
2 MOOC
6 MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC
13 MOOC, MOOC, MOOC
14 MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC, MOOC
-16 MOOC
35 MOOC
55 IRM
105 MOOC, MOOC, MOOC
-800 IRM, IRM, IRM, IRM, IRM, IRM, IRM, IRM, IRM, IRM, IRM
-801 IRM, IRM, IRM
-867 IRM, IRM
-3132 Java
-4285 MOOC
-4270 MOOC
-4305 Java
-4306 MOOC
-4311 MOOC
-4312 Java
-4313 MOOC
-4325 MOOC
-4348 MOOC
-4579 MOOC
-4580 Acnet-NG
-4595 MOOC
-4597 MOOC
-4617 MOOC
-4621 MOOC
-4994 MOOC
-12000 MOOC
-15000 MOOC
-25999 MOOC
-73000 MOOC
132000 MOOC
1941268 IRM
-637064795 MOOC
-637064820 MOOC
-981885986 MOOC, MOOC, MOOC
-1572005922000 MOOC
-1600958660000 MOOC

History

#1 Updated by Dennis Nicklaus about 2 months ago

SRFLPR (Rich) and BLLRFD (Pellico) currently show system dates in 1970. I didn't see any obvious attempts at periodic time updating/syncing in their startup scripts.

Also available in: Atom PDF