IOTA BPM Front End¶
About this Document¶
This document is maintained as a Wiki page in the AD Instrumentation / adinstbpm Redmine at https://cdcvs.fnal.gov/redmine/projects/adinstbpm/wiki/IOTAFE
Please refer to the above URL for the most recent revision to this document.
This document describes the technical design of the IOTA BPM Front End software. The IOTA BPM Front End is a single VME crate containing an x86 Single Board Computer (SBC), timing and clock decoder module and data acquisition (DAQ) modules. The SBC runs a custom Linux system built with Buildroot, data acquisition software and interfaces to the Fermilab Accelerator Controls System (ACNET) via the AD/Controls ACSys/FE framework.
Hardware Components¶The operational hardware for the IOTA BPM Front End consists of:
- Single Weiner VME Crate
- Remote reboot and monitoring capability via TCP/IP, supported through ACNET and web interface
- 1x Concurrent 405x SBC
- Intel Core Duo @ 2.0 GHz
- 2 GB Ram
- Linux v3.16 kernel
- ACSys/FE Framework
- 1x IOTA Timing and Clock Decoder Module
- 10x IOTA BPM Data Acquisition Modules
- VME BLT cycles - 64 Mb/s
- Option to use GigE ethernet with switch - 1 Gb/s
A test stand with the same hardware is maintained in TGS-100.
Concurrent 405x SBC¶
The Concurrent 405x SBC is a commercial off-the-shelf (COTS) Intel-based VME SBC produced by Concurrent Technologies. This SBC is equipped with an Intel Core Duo, a dual core 64-bit x86 processor and 2GB of system RAM. The front panel provides 2x Gigabit ethernet (GigE) interfaces. One of these interfaces is reserved for communicating with ACNET through the controls system LAN. The second interface could be used for communicating with the DAQ module's over a private GigE LAN (this would require adding a GigE switch to the system). Storage is provided through a removable 2GB compact flash disk. A single PMC or XMC expansion site is provided with an available PCIe 1x lane.
IOTA Timing and Clock Decoder Module (IOTATime)¶
This is an in-house developed 2-slot VME module. The module is responsible for delivering clock and trigger signals to the DAQ modules.
Supports multiple trigger inputs. Provides DAQ modules with a 'trigger ID' identifying which trigger event is being broadcast. The DAQ modules use this ID to select which data acquisition configuration to use. This alleviates the need for the CPU to program the configuration to each DAQ module during the arming sequence.
Decodes NML TCLK events and provides interrupts to CPU. This allows us to synchronize the data acquisition software with IOTA machine events.
The software interface is provided via VME A16/D16 registers.
250MHz 8-channel IOTA BPM Data Acquisition Module (IOTABPM250x8)¶
This is an in-house developed 250MHz 8-channel VME-based data acquisition module with 256MB of on-board RAM.
Data transfers are supported via VME BLT cycles with an observed throughput of 64Mb/s. The module also has a GigE interface on the front panel that could support transfers via UDP or TCP over a private GigE LAN. NOTE: the GigE interface is still under development.
Buildroot Embedded Linux OS¶
Buildroot is an open-source project that provides a menu-driven tool for building complete Linux systems (kernel plus root file system) from source code. Buildroot is targeted towards building low foot-print embedded systems as small as 2MB. For comparison, most desktop Linux distributions require an installation foot-print exceeding 2 gigabytes. To achieve this goal Buildroot replaces the standard GNU C library (libc) with a slimmed-down alternative, ucLibc. Also, the shell and most of the common command-line utilities are provided through a single program called Busybox. In addition to building the kernel and root filesystem, Buildroot also downloads and builds the cross-compile tool-chain for building software to run on the embedded target.
The small foot-print of the root file system allows us to "piggy back" the OS support packages inside of the kernel and too load the entire system into RAM at boot. This speeds up performance, simplifies software upgrades and reduces the write cycles to the compact flash (extending it's lifetime).
The Linux kernel selected for the IOTA BPM system is version 3.16. This is recent enough to include support for the VME bus (earlier versions of the Linux kernel only support VME through third party modules). The support for the VME bus and the VME bridge chips currently on the market is active and stable, with commits to drivers/vme as recently as September 2018. Upgrading to more recent kernels is possible but would require some effort to port kernel-level code that is dependent on older APIs.
Buildroot configurations for the Concurrent 405x SBC target are maintained as a part of the ees-buildroot project.
IOTATime is a Linux kernel module and an application layer C++ library for interfacing with the IOTA Timing and Clock Decoder module.
The Linux kernel module (LKM) is called iotatimemod and interfaces directly with the IOTA Timing and Clock Decoder hardware module via the Linux VME API. The VME bus is scanned for the presence of the hardware when the LKM is inserted at boot time. Output is generated to the serial console and the system log file to announce the discovery of the hardware or any errors encountered during the scan.
If the hardware module is detected then the LKM will register a VME device with the Linux device model. A new entry will appear under the VME bus in the sysfs tree (mounted at /sysfs). To assist with debugging, all of the IOTATime registers are exposed to the user via the sysfs tree. The LKM also attaches an interrupt handler to VME interrupt line 5 with vector 0x10. However, the device is not programmed to generate interrupts until the user requests the device be initialized and provides TCLK events to be decoded.
Access to the hardware is provided to user space via the traditional Linux system calls and sysfs. Interrupt notification is provided through the Linux Generic Netlink API. A C++ library, libiotatime is provided for the application layer to interface with IOTATime devices. This library wraps these APIs and presents the user with a clean object-oriented interface.
The IOTATime driver was developed with the help of the drvgen tool. With drvgen, a specification is given as a spreadsheet that describes the hardware's register map, interrupt lines, memory areas and commands. The tool then generates boiler-plate code for reading/writing the registers, scanning for hardware, registering interrupts and facilitating these requests between user-space and the LKM. Customization proceeds through hooks and sub-classes, so if the specification changes the generated code can be re-generated without writing over the custom code. The specification and source code reside in the AD Instrumentation Drivers project.
IOTABPM250x8 is a Linux kernel module and an application layer C++ library for interfacing with the 250Mhz 8-channel IOTA BPM data acquisition module.
The implementation of IOTABPM250x8 and the interfaces between it's LKM and application library are similar to IOTATime and won't be rehashed here. One important feature of IOTABPM250x8 that is not present in IOTATime is support for VME block-transfers (BLT). VME BLT allows data to be transferred from the data acquisition modules more efficiently over the VME bus and with the help of a DMA controller. This alleviates the demands placed on the CPU when reading out large amounts of data, allowing it to service other tasks.
The data acquisition modules provide an interrupt to the CPU after data has been acquired. The LKM registers an interrupt handler on VME interrupt line 4 with vectors starting at 0x20 to service these notifications. However, the data acquisition software only enables the first module to provide the interrupt as enabling it on all would be redundant.
As with the IOTATime driver, the IOTABPM250x8 driver was developed with the help of the drvgen tool. The specification and source code reside in the AD Instrumentation Drivers project.
The coordination of the data acquisition modules, trigger signals, machine clock events and data buffers is handled by the bpmd daemon process, the major component of the adinstbpm project. Originally developed for the Booster BPM upgrade, this component is designed to be adaptable to multiple BPM systems. The IOTA BPM system is the second deployment of a BPM system based on adinstbpm. Development has proceeded somewhat in parallel with the Booster BPM since the summer of 2018.
The architecture of the IOTA BPM Front End software is described in the figure below. The basis for the architecture is a Blackboard pattern, where the software components organize themselves around a shared memory region that contains data from the problem domain (in this case the data from the data acquisition modules). The components that make up the adinstbpm project are bpmd, bpmcli and IOTA ACNET, each shown as a separate process in the diagram. The processes communicate with each other via the shared memory region and a message queue (labeled as the "Control Queue"). The ITOABPM250x8 and IOTATime LKMs are shown on the kernel side of the diagram. The bpmd process interfaces to the LKMs using the application-layer libraries they provide. The LKMs communicate with the hardware via the Linux VME APIs.
The bpmd process is responsible for managing the shared memory region, coordinating the data acquisition modules with the machine state and initiating the readout of the BPM data.
Data is readout from the data acquisition modules at the end of a measurement and is stored directly into a buffer inside of the shared memory region. Multiple processes have access to the shared memory region but only the data acquisition LKM has write access. Mutual exclusion is guaranteed on the buffer-level using POSIX shared semaphores. Position calculations are deferred until requested by the user through ACNET, Backdoor or bpmcli.To adapt adinstbpm to IOTA, a couple of specialized components were developed -
- A state machine for IOTA, which programs the IOTATime hardware to respond to IOTA TCLK events and to send the appropriate triggers to the data acquisition modules.
- Adapters for the IOTATime and IOTABPM250x8 hardware. These are classes which follow the Adapter pattern and wrap the device objects provided by libiotatime and libiotabpm250x8 and 'adapt' them to an interface expected by adinstbpm.
- An ACNET interface according to specifications from IOTA experts
- Tells the process to load the IOTA state machine and IOTA device drivers.
- Describes all of the BPMs, their channel configurations, timing delays and scaling.
- Measurements and data acquisition profiles
- Which machine cycles to respond to and when to perform readouts
Backdoor is a network protocol developed in-house for communicating with Front Ends without going through ACNET. It has bindings for Labview and MATLAB which make it an ideal choice for developing "engineering" applications with Graphical User Interfaces (GUIs). A server process runs on the IOTA BPM front end for facilitating the communication with clients. A custom LKM has been developed which exposes the VME bus via Backdoor and allows an expert to use the GUIs to see how the hardware is behaving during operation.
Backdoor accessors and a Labview GUI are under development which will provide access to the BPM data buffers.
The primary diagnostic tool for adinstbpm is a command-line utility called bpmcli. The bpmcli utility allows an expert to examine the data stored in the shared memory region and to send commands to the bpmd process through the control message queue. All of the data structures in the shared memory region and all of the commands are implemented. The operation of bpmcli is documented in a wiki page as well as through an online 'help' function.
ACSys/FE is a software framework provided by AD/Controls which assumes the role of coordinating with ACNET (think of it as MOOC for Linux systems). This framework is developed in Erlang but provides developers with a C++ interface via a "driver" called cdev. The cdev interface starts up your C++ interface as a separate process and facilitates communication with ACSys/FE through traditional Linux IPC.The IOTA ACNET interface is implemented as two cdev classes -
- iotacrate_acsys - Handles ACNET devices that operate on the front end as a whole or return data for multiple BPMs through a single device (e.g. orbit devices)
- iotabpm_acsys - Handles ACNET devices that operate on per-BPM basis (e.g. turn-by-turn data for a specific BPM). In this case a separate iotabpm_acsys process is launched for each BPM. This allows requests for different BPMs to be handled concurrently, making ACNET communication more efficient when dealing with large requests.
The ACNET interfaces read their data from the shared memory region and present it in a format suitable for consumption by the operator. Commands are communicated to bpmd via the control queue and status is returned via the shared memory. Because the ACNET interfaces run as separate processes they can be restarted and upgraded at will without affecting data collection. This architecture also localizes faults and makes the system more robust.
The IOTA BPM user interfaces are currently implemented as a standard parameter page (N23) and via Synoptic pages developed by FAST/IOTA machine experts.