The DAQ system for the PET-TOF project consists of a 32 channel 5GS/s VME digitizer board inside a DAWN VME create and an Intel VME processor board running linux. The hostname of the processor is vmelx1.dhcp.fnal.gov.
The digitizer can be accessed in A32 and A24 space. It is currently configured to reside at 0x0 in A32 and A24 space.
The host processor is a GE Fanuc VMIVME7750 running a stripped down version of SLF 3 with a kernel.org kernel, version 2.6.15. The kernel includes TRACE but is otherwise stock. The operating system and digitizer readout software are located on a 64MB compact flash card that the processor boots off of. The processor sits on the FNAL public network and retrieves it's IP address via DHCP with the vmelx1.dhcp.fnal.gov hostname. A kerberized ssh daemon is running and allows anyone with an account on the machine to login. It is also possible to login remotely via a keyboard and monitor, but only as root.
Note that the kernel, distribution and ssh daemon on this machine are relatively old. It's not possible to ssh to this machine from newer systems as there incompatibilities between the ssh client on newer machines and the ssh daemon on the host processor. Logins from fnalu seem to work fine.
VME Kernel Module¶
The version of the kernel that is running on the host processor does not include VME support. I looked into newer kernels that do include VME support but couldn't get anything to work. The VME support in newer kernels is also limited in that it doesn't support DMA or interrupts. The "GE Fanuc Universe VMEbus Driver for Linux" which is distributed outside the kernel is being used. The driver comes with a library that simplifies access to the VME bus which is also being used. The source code for the driver and the library is included in the git repository.
The host processor is running a 32 bit version of linux. None of the development machines I had access to had a 32 bit compiler that was able to compile the older kernels. I used Crosstool to build a 32bit cross compiler. The cross compiler used gcc v4.1.1 and glibc 2.3.6. The cross tool source code is also included in the git repository as it's old and doesn't seem to be active anymore.
Digitizer Configuration/Readout Software¶
The software to configure and control and digitizer is a single C module named v1742.c. It is installed in /bin on the host processor. The software is maintained in the pet-tof git repository.
If no command line options are specified the board defaults to sampling at 5GS/s with 1024 samples per trigger.There are several registers that are hard coded and not modifiable via the command line parameters:
- Trigger Source Enable Mask (0x810C) - Software and external triggers are enabled.
- Trigger Out Enable (0x8110) - All external triggers are disabled.
- Group Enable (0x8120)- All groups are enabled.
- TR0 DC Offset (0x10D4) - Signal offset for group 0 and 1 trigger. Set for NIM logic levels.
- TR0 Threshold (0x10DC) - Trigger threshold for group 0 and 1 trigger. Set for NIM logic levels.
- TR1 DC Offset (0x12D4) - Signal offset for group 2 and 3 trigger. Set for NIM logic levels.
- TR2 Threshold (0x12DC) - Trigger threshold for group 2 and 3 trigger. Set for NIM logic levels.
- Group Config (0x8000) - The low latency trigger is enabled and is set to trigger on the rising edge. The other options are set via the command line.
Command Line Options¶
The digitizer software will print out the command line help if no options are specified or if an invalid option is specified. It is possible change the sample rate, the number of samples per trigger, whether or not to digitize the trigger from the command line and whether or not the FPGA on the digitizer will inject test data into the sample memory.
Command line parameters: --status Display the status of the digitizer board. --run Configure the board to acquire data and read out data once it is acquired. --test-mode Replace ADC samples with a sawtooth generated by the FPGA. --digitize-trigger Digitize the TR0 and TR1 signals and include them in the output data. --sample-rate Set the sample rate. Can be 5, 2.5 or 1. --num-samples Set the number of samples to take per trigger. Can be 1024, 520, 256 or 136. --software-trigger Trigger an acquistion with the software trigger. --split-data [datafile] Read in the raw data from the digitizer and split it into individual files for each channel. The metadata is removed and each file only contains the samples for that channel.
The data format is specified in the manual in section 3.6. The application will read the data verbatim out of the digitizer memory and store it in the current working directory. Each file is timestamped with the number of seconds since the unix epoch. The application can convert the digitizer data format into something that is easier to parse. The "--split-data" parameter will take raw digitizer data and split it into individual files for each channel. The individual files only contain sample data.
The current plan is to mount via NFS the disk of another machine in the lab for storage. This will work fine as the data rate and the amount of data is small.