Project

General

Profile

Support #9801

Assist with debugging SSP network issues

Added by Kurt Biery about 5 years ago. Updated about 5 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Start date:
08/07/2015
Due date:
% Done:

0%

Estimated time:
Duration:

Description

Recently, there were problems observed with data transfers out of SSPs. In response to that, the network switch that connects to the SSPs was replaced, and that caused even more problems. There is a ServiceNow ticket for this issue (https://fermi.service-now.com/nav_to.do?uri=sc_req_item.do?sys_id=755d35636fe94240d6efce026e3ee4bd%26sysparm_view=ess), but I'm also creating this Redmine Issue with various notes for historical purposes.

History

#1 Updated by Kurt Biery about 5 years ago

Additional Information: email from Ron to Geoff:

Hi Geoff, all,

Geoff, thanks for your help with the switch and the nudge toward wireshark.

As some may know, I've known intimately wireshark's protocol dissection
capabilities as I was the one that initialed the CA (Channel Access) dissection
currently available for wireshark. However, I was unaware of the analysis-filtering
capability.

So, I thought I would pass along a few bits of info which I know will come in very
handy in the future.

As an example, I did an scp of a large file between 2 systems and during the
transfer I momentarily disconnected the wire. This, of course, would cause
retransmission to occur. (In a "many to one" (bottleneck) daq system trying to
achieve max rate, retransmissions should not occur; although they prevent
data lose, they are a detriment to rate. Window size tuning can eliminate
the need for retransmissions!)

I can easily ask wireshark/tshark if there was
retransmissions as follows (for the case when I don't know for sure that they
occurred or not):

  1. # Collect some data...
  2. tcpdump ieth0 -n -s64 -wt.tcpdump host fnapcf
    tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 64 bytes
    ^C193398 packets captured
    193398 packets received by filter
    0 packets dropped by kernel
    --08/07_10:53:06-
  1. # Ask tshark "was there retransmissions in this data file?"
  2. tshark rt.tcpdump tcp.analysis.retransmission
    Running as user "root" and group "root". This could be dangerous.
    40151 18.410598 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    40167 20.034107 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    73420 22.126922 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    172605 28.217794 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    172618 28.218108 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Fast Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    172627 30.054077 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    172629 30.054344 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    172631 30.054396 131.225.84.23 -> 10.0.2.15 SSH 1474 [TCP Retransmission] Encrypted response packet len=1420[Packet size limited during capture]
    --08/07_10:54:28-

PLEASE NOTE:
1) I used tcpdump to capture the data between wireshark/tshark have an inefficiency in the capture
that caused dropped packets for the capture (no impact on actual data between the 2 systems!).
2) the data file can easily be loaded into the graphical wireshark application.

Thanks again Geoff.

Cheers,
Ron

Also available in: Atom PDF