Support #9801
Assist with debugging SSP network issues
0%
Description
Recently, there were problems observed with data transfers out of SSPs. In response to that, the network switch that connects to the SSPs was replaced, and that caused even more problems. There is a ServiceNow ticket for this issue (https://fermi.service-now.com/nav_to.do?uri=sc_req_item.do?sys_id=755d35636fe94240d6efce026e3ee4bd%26sysparm_view=ess), but I'm also creating this Redmine Issue with various notes for historical purposes.
History
#1 Updated by Kurt Biery over 5 years ago
Additional Information: email from Ron to Geoff:
Hi Geoff, all,
Geoff, thanks for your help with the switch and the nudge toward wireshark.
As some may know, I've known intimately wireshark's protocol dissection
capabilities as I was the one that initialed the CA (Channel Access) dissection
currently available for wireshark. However, I was unaware of the analysis-filtering
capability.
So, I thought I would pass along a few bits of info which I know will come in very
handy in the future.
As an example, I did an scp of a large file between 2 systems and during the
transfer I momentarily disconnected the wire. This, of course, would cause
retransmission to occur. (In a "many to one" (bottleneck) daq system trying to
achieve max rate, retransmissions should not occur; although they prevent
data lose, they are a detriment to rate. Window size tuning can eliminate
the need for retransmissions!)
I can easily ask wireshark/tshark if there was
retransmissions as follows (for the case when I don't know for sure that they
occurred or not):
- # Collect some data...
- tcpdump
ieth0 -n -s64 -wt.tcpdump host fnapcf
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 64 bytes
^C193398 packets captured
193398 packets received by filter
0 packets dropped by kernel
--08/07_10:53:06-
- # Ask tshark "was there retransmissions in this data file?"
- tshark
rt.tcpdump tcp.analysis.retransmission
Running as user "root" and group "root". This could be dangerous.
40151 18.410598 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
40167 20.034107 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
73420 22.126922 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
172605 28.217794 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
172618 28.218108 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
172627 30.054077 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
172629 30.054344 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
172631 30.054396 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
--08/07_10:54:28-
PLEASE NOTE:
1) I used tcpdump to capture the data between wireshark/tshark have an inefficiency in the capture
that caused dropped packets for the capture (no impact on actual data between the 2 systems!).
2) the data file can easily be loaded into the graphical wireshark application.
Thanks again Geoff.
Cheers,
Ron