IpBus information and lessons learned

Information here mainly comes for the following docmeunts:

And from the experiences of users at UCL (Firmware: Erdem, Matt. Software: Mark, Tom).

Quick intro to IpBus packets

Three are three types of IpBus packets:

  • Control Packets
  • Status Packets
  • Resend Packets

Each type has an identifying code that is included in the IpBus packet header. The header also includes:

  • packet ID
  • endian-ness of packet
  • IpBus protocol version

See the following brief descriptions of the packet types...

Control packets

This is a packet header along with a number of transactions. Consecutive control packets must have consecutive packet IDs or else it is (silently) dropped (unless the ID is 0, which is always accepted).

The response to a control packet must contain the same header as the initial control packet (as a reliability check).

Each transaction itself has a header containing:
  • transaction ID
  • number of words following header
  • type ID (e.g. read, write)
  • info code (used to register success/error codes etc)
  • IpBus protocol version

The transaction request will then have some following words, such as register address, and a value if the request is a write command.

The transaction response will contain an identical header (except for the info code, which indicates success/failure of request), and any resulting response data (such as the register values for a read request).

Note that a client may continue sending multiple control packets to a target even if no response has yet been received. As long as the targets buffer is not overrun (can be checked via status packet), it will continue to queue the requests and transmit the responses as they are available. The result of the request will depend on the state of the target at the time the request is processed, not when it is received. So for example the response to a read request is the value at the time the request is processed, not the time the request was received (and added to the queue).

Status packets

A status request is used to obtain the recent history of the target's activity, including recent incoming and outgoing packets, and status messages. It also checks information about the target, such as its buffer size for holding queued control packets.

It is used in the reliability mechanism, and also for debugging purposes.

The request packet contains a simple header (including packet ID) and a number of empty (=0) reserved words.

The response contains the identical header, information on the last 16 packets and the headers from the last 8 packets.

Resend packets

These packets request a resend of recent output control packets from the target. If the transaction ID for the control packet specified is still in the target's outbound buffer, then the outbound packet will be resent to the client.

The packet header contains the ID of the control packet that should be resent. Other that the packet type code (which indicates "resend" rather than "control" packet), it is identical to the control packet header for the control packet that should be resent.

Reliability mechanism

UDP is not a reliable protocol, but IpBus implements its own reliability mechanism. This means that if a packet is lost (in either direction, e.g. a request or a response), the client can detect this and request a resend. This relies on the fact that consecutive packs have consecutive packet IDs, and hence both the client and target are able to detect if the sequence is broken.

If a request is lost, the target identifies an unexpected packet ID on the next request and stops. The client then detects a timeout, and requests a status packet from the target. This shows the last response transmitted, and the client sees that it did not receive it, and requests a resend.

If a response is lost, the client either times out waiting for it or identifies an unexpected packet ID from a subsequent response (whichever comes first). The client can then check via a status request that the response was transmitted, and requests a resend.

In both cases the client is responsible for requesting a resend.

TODO What is not clear from documentation is whether this is automatically handled in the client software (e.g. the HWManager, COnnectionManager etc) when command packets are sent, or if this needs to be implemented in SW. I think it is probably done automatically by IpBus under the bonnet, but it doesn't seem to be doing this on our ystem. Perhaps we are not using up-to-date enough versions, as this seems to be new feature in IpBus 2.X.X.

Note that this only works if there is a single point of contact for the target, sp if multiple clients are required then a control hub must be used to ensure a single unbroken packet numbering system.

IpBus network overview

In IpBus, the two main network nodes are:

  • clients - send requests and receive responses (e.g. some control software)
  • targets - receive requests and send responses (e.g. the firmware of a DAQ device)

A client may send requests and receive responses from multiple targets.

If multiple clients are used to control a given target, the traffic should be routed through a control hub. If a control hub is used, the IpBus reliability system will (ironically) not be reliable.

The current g-2 test setup is:

  • client = (which itself is instantiated either by the MIDAS frontend of the standalone python GUI
  • target = ATLYS board (which is connected to TDCs, which themselves are connected to ASDQs and then the straws)
  • the client-target relationship is 1-1, and no hub is used
  • UDP is chosen as the packet protocol

Dispatching IpBus packets

When individual IpBus transaction requests are invoked (e.g. using hw->getNode(node_id).read(), hw->getNode(node_id).write(val) etc), these are added to the next IpBus packet. This is not sent however until hw.dispatch() is called. When dispatch() is called, the full IpBus packet is sent containing all the built up individual IpBus transactions.

As some amount of additional headers (from IpBus, UDP, IP, Ethernet etc) are added to these packets, it is more efficient to send multiple transaction requests in a single packet, rather than calling dispatch() after every transaction request (e.g. if you are sending 3 read requests, it is more efficient to do read1,read2,read3,dispatch rather than read1,dispatch,read2,dispatch,read3,dispatch etc). Think about this if performance is a consideration.

Note that the returned result of a read transaction will not be available until after the dispatch is called (as the response returned over the network). Therefore an error (non-validated memory) will be received if the value of a ValWord or ValVector instance created to accept a read result is used prior to dispatch. See below for example:

ValWord<uint32_t> val = hw->getNode(reg).read();
//Attempts to use val or val.value() here will throw exception
//Attempts to use val or val.value() here are fine

Note also that there is a maximum limit to the size of an ethernet packet, and after the various headers are subtracted this results in the maximum IpBus packet size is 368 32bit words. You therefore need to ensure that the number of transactions you send does not result in larger returned packets than this (either in the original transaction or the response), and larger block transfers in particular need to be handled at software level.

Information source: