Project

General

Profile

IpBus information and lessons learned » History » Version 1

Thomas Stuttard, 08/25/2016 05:19 PM

1 1 Thomas Stuttard
h1. IpBus information and lessons learned
2 1 Thomas Stuttard
3 1 Thomas Stuttard
Information here mainly comes for the following docmeunts:
4 1 Thomas Stuttard
5 1 Thomas Stuttard
* http://ohm.bu.edu/~chill90/ipbus/ipbus_protocol_v2_0.pdf
6 1 Thomas Stuttard
* https://svnweb.cern.ch/trac/cactus/wiki/uhalQuickTutorial#HowtoInstalltheIPbusSuite
7 1 Thomas Stuttard
8 1 Thomas Stuttard
And from the experiences of users at UCL (Firmware: Erdem, Matt. Software: Mark, Tom).
9 1 Thomas Stuttard
10 1 Thomas Stuttard
h2. Quick intro to IpBus packets
11 1 Thomas Stuttard
12 1 Thomas Stuttard
Three are three types of IpBus packets:
13 1 Thomas Stuttard
14 1 Thomas Stuttard
* *Control Packets*
15 1 Thomas Stuttard
* *Status Packets*
16 1 Thomas Stuttard
* *Resend Packets*
17 1 Thomas Stuttard
18 1 Thomas Stuttard
Each type has an identifying code that is included in the IpBus packet header. The header also includes:
19 1 Thomas Stuttard
20 1 Thomas Stuttard
* packet ID
21 1 Thomas Stuttard
* endian-ness of packet
22 1 Thomas Stuttard
* IpBus protocol version
23 1 Thomas Stuttard
24 1 Thomas Stuttard
See the following brief descriptions of the packet types...
25 1 Thomas Stuttard
26 1 Thomas Stuttard
h3. Control packets
27 1 Thomas Stuttard
28 1 Thomas Stuttard
This is a packet header along with a number of transactions. Consecutive control packets must have consecutive packet IDs or else it is (silently) dropped (unless the ID is 0, which is always accepted).
29 1 Thomas Stuttard
30 1 Thomas Stuttard
The response to a control packet must contain the same header as the initial control packet (as a reliability check). 
31 1 Thomas Stuttard
32 1 Thomas Stuttard
Each transaction itself has a header containing:
33 1 Thomas Stuttard
* transaction ID
34 1 Thomas Stuttard
* number of words following header
35 1 Thomas Stuttard
* type ID (e.g. read, write)
36 1 Thomas Stuttard
* info code (used to register success/error codes etc)
37 1 Thomas Stuttard
* IpBus protocol version
38 1 Thomas Stuttard
39 1 Thomas Stuttard
The transaction request will then have some following words, such as register address, and a value if the request is a write command.
40 1 Thomas Stuttard
41 1 Thomas Stuttard
The transaction response will contain an identical header (except for the info code, which indicates success/failure of request), and any resulting response data (such as the register values for a read request).
42 1 Thomas Stuttard
43 1 Thomas Stuttard
Note that a client may continue sending multiple control packets to a target even if no response has yet been received. As long as the targets buffer is not overrun (can be checked via status packet), it will continue to queue the requests and transmit the responses as they are available. The result of the request will depend on the state of the target at the time the request is processed, not when it is received. So for example the response to a read request is the value at the time the request is processed, not the time the request was received (and added to the queue).
44 1 Thomas Stuttard
45 1 Thomas Stuttard
h3. Status packets
46 1 Thomas Stuttard
47 1 Thomas Stuttard
A status request is used to obtain the recent history of the target's activity, including recent incoming and outgoing packets, and status messages. It also checks information about the target, such as its buffer size for holding queued control packets.
48 1 Thomas Stuttard
49 1 Thomas Stuttard
It is used in the reliability mechanism, and also for debugging purposes.
50 1 Thomas Stuttard
51 1 Thomas Stuttard
The request packet contains a simple header (including packet ID) and a number of empty (=0) reserved words.
52 1 Thomas Stuttard
53 1 Thomas Stuttard
The response contains the identical header, information on the last 16 packets and the headers from the last 8 packets.
54 1 Thomas Stuttard
55 1 Thomas Stuttard
h3. Resend packets
56 1 Thomas Stuttard
57 1 Thomas Stuttard
These packets request a resend of recent output control packets from the target. If the transaction ID for the control packet specified is still in the target's outbound buffer, then the outbound packet will be resent to the client.
58 1 Thomas Stuttard
59 1 Thomas Stuttard
The packet header contains the ID of the control packet that should be resent. Other that the packet type code (which indicates "resend" rather than "control" packet), it is identical to the control packet header for the control packet that should be resent. 
60 1 Thomas Stuttard
61 1 Thomas Stuttard
h2. Reliability mechanism
62 1 Thomas Stuttard
63 1 Thomas Stuttard
UDP is not a reliable protocol, but IpBus implements its own reliability mechanism. This means that if a packet is lost (in either direction, e.g. a request or a response), the client can detect this and request a resend. This relies on the fact that consecutive packs have consecutive packet IDs, and hence both the client and target are able to detect if the sequence is broken.
64 1 Thomas Stuttard
65 1 Thomas Stuttard
If a request is lost, the target identifies an unexpected packet ID on the next request and stops. The client then detects a timeout, and requests a status packet from the target. This shows the last response transmitted, and the client sees that it did not receive it, and requests a resend.
66 1 Thomas Stuttard
67 1 Thomas Stuttard
If a response is lost, the client either times out waiting for it or identifies an unexpected packet ID from a subsequent response (whichever comes first). The client can then check via a status request that the response was transmitted, and requests a resend.
68 1 Thomas Stuttard
69 1 Thomas Stuttard
In both cases the client is responsible for requesting a resend.
70 1 Thomas Stuttard
71 1 Thomas Stuttard
*TODO What is not clear from documentation is whether this is automatically handled in the client software (e.g. the HWManager, COnnectionManager etc) when command packets are sent, or if this needs to be implemented in SW. I think it is probably done automatically by IpBus under the bonnet, but it doesn't seem to be doing this on our ystem. Perhaps we are not using up-to-date enough versions, as this seems to be  new feature in IpBus 2.X.X.*
72 1 Thomas Stuttard
73 1 Thomas Stuttard
Note that this only works if there is a single point of contact for the target, sp if multiple clients are required then a control hub must be used to ensure a single unbroken packet numbering system.
74 1 Thomas Stuttard
75 1 Thomas Stuttard
h2. IpBus network overview
76 1 Thomas Stuttard
77 1 Thomas Stuttard
In IpBus, the two main network nodes are:
78 1 Thomas Stuttard
79 1 Thomas Stuttard
* clients - send requests and receive responses (e.g. some control software)
80 1 Thomas Stuttard
* targets - receive requests and send responses (e.g. the firmware of a DAQ device)
81 1 Thomas Stuttard
82 1 Thomas Stuttard
A client may send requests and receive responses from multiple targets.
83 1 Thomas Stuttard
84 1 Thomas Stuttard
If multiple clients are used to control a given target, the traffic should be routed through a control hub. If a control hub is used, the IpBus reliability system will (ironically) not be reliable.
85 1 Thomas Stuttard
86 1 Thomas Stuttard
The current g-2 test setup is:
87 1 Thomas Stuttard
88 1 Thomas Stuttard
* client = StrawDAQ.cc (which itself is instantiated either by the straws_frontend.cc MIDAS frontend of the standalone python GUI Gm2RunControl.py)
89 1 Thomas Stuttard
* target = ATLYS board (which is connected to TDCs, which themselves are connected to ASDQs and then the straws)
90 1 Thomas Stuttard
* the client-target relationship is 1-1, and no hub is used
91 1 Thomas Stuttard
* UDP is chosen as the packet protocol
92 1 Thomas Stuttard
93 1 Thomas Stuttard
h2. Dispatching IpBus packets
94 1 Thomas Stuttard
95 1 Thomas Stuttard
When individual IpBus transaction requests are invoked (e.g. using hw->getNode(node_id).read(), hw->getNode(node_id).write(val) etc), these are added to the next IpBus packet. This is not sent however until hw.dispatch() is called. When dispatch() is called, the full IpBus packet is sent containing all the built up individual IpBus transactions.
96 1 Thomas Stuttard
97 1 Thomas Stuttard
As some amount of additional headers (from IpBus, UDP, IP, Ethernet etc) are added to these packets, it is more efficient to send multiple transaction requests in a single packet, rather than calling dispatch() after every transaction request (e.g. if you are sending 3 read requests, it is more efficient to do _read1,read2,read3,dispatch_ rather than _read1,dispatch,read2,dispatch,read3,dispatch_ etc). Think about this if performance is a consideration.
98 1 Thomas Stuttard
99 1 Thomas Stuttard
Note that the returned result of a read transaction will not be available until after the dispatch is called (as the response returned over the network). Therefore an error (non-validated memory) will be received if the value of a ValWord or ValVector instance created to accept a read result is used prior to dispatch. See below for example:
100 1 Thomas Stuttard
101 1 Thomas Stuttard
<pre>
102 1 Thomas Stuttard
ValWord<uint32_t> val = hw->getNode(reg).read();
103 1 Thomas Stuttard
//Attempts to use val or val.value() here will throw exception
104 1 Thomas Stuttard
hw->dispatch();
105 1 Thomas Stuttard
//Attempts to use val or val.value() here are fine
106 1 Thomas Stuttard
</pre>
107 1 Thomas Stuttard
108 1 Thomas Stuttard
Note also that there is a maximum limit to the size of an ethernet packet, and after the various headers are subtracted this results in the maximum IpBus packet size is 368 32bit words. You therefore need to ensure that the number of transactions you send does not result in larger returned packets than this (either in the original transaction or the response), and larger block transfers in particular need to be handled at software level.
109 1 Thomas Stuttard
110 1 Thomas Stuttard
*Information source:* http://ohm.bu.edu/~chill90/ipbus/ipbus_protocol_v2_0.pdf