Mu2e sync demo » History » Version 106

« Previous - Version 106/236 (diff) - Next » - Current version
Gregory Rakness, 01/11/2019 04:58 PM

Mu2e sync demo


With the demo, we want to demonstrate that...
  • Using loopback, Event Windows are synchronized with ROC-to-ROC relative synchronization < 5 ns
    • In other words, t=0 at one ROC is within 5 ns of t=0 at any other ROC
    • 5ns comes from the granularity with which the DAQ FPGA sends and transmits data over the optical fiber (200MHz)
  • With resets, power cycles, beginning of run, etc., event-to-event jitter in the relative synchronization is < 500ps standard deviation
    • In other words, once the Event Windows are synchronized, the value of the relative synchronization between ROCs is fixed
  • Using fine phase delay, make fine adjustments to timestamp the data into ROC
    • Timestamp the data with < 1ns granularity coming into ROC

These goals can be achieved by completing the following:

  • Establish loopback to measure fiber length
    • Make distribution of measured times for the loopback: CFO -> DTC -> ROC -> DTC -> CFO (distribution is made in software, not in CFO firmware)
    • The "loopback distribution" should have width 2x 5-ns-wide bins
    • The average value of the loopback distribution should be reproducible (e.g., after power cycles) within 500ps
    • The average value of the loopback distribution has contributions from the latencies of the receive and transmit at each component, latency in the loopback at the ROC, and propagation time proportional to (2 * fiber length)
    • A plot of loopback dependence on fiber length would be a sanity check that the loopback is doing something sensible
  • Timestamp data at ROC and apply delays to synchronize
    • ROC firmware receive external data (CAPTAN pulse) and timestamp it. The timestamp is based on the Event Window start (received from the CFO)
    • ROC firmware apply coarse (step = about 5ns) and fine delays (fine delay step << 5 ns) to modify timestamp of data
  • Measure ROC-to-ROC relative synchronization
    • Connect 2 parallel CFO-ROC (clock/event window) connections from the same CFO: CFO -> DTC_1 -> ROC_1 and CFO -> DTC_2 -> ROC_2
    • Connect 2 parallel CAPTAN-ROC (external data) connections from the same CAPTAN: CAPTAN -> ROC_1 and CAPTAN -> ROC_2, with equal cable lengths to guarantee synchronous arrival time of CAPTAN pulse.
    • Make distribution of timestamp difference. Compute for each event: (timestamp_2 - timestamp_1), make plot. (distribution is made in software)
    • The "timestamp-difference distribution" should populate 3x bins (5 ns-wide bins): (N-1, N, N+1)
      • Changing coarse relative delay between ROC_1 and ROC_2 will the change value of N -> should be able to make N=0 by applying loopback delay
      • Changing fine relative delay between ROC_1 and ROC_2 will change the population of the 3x bins -> the amount that this knob needs to be changed to synchronize ROC_1 and ROC_2 gives the resolution of the loopback

Operate the system

Startup otsdaq

Use otsdaq to operate the system

set of commands:
  • kinit -f
  • ssh mu2edaq01
  • ssh mu2edaq06
  • ksu mu2edaq
  • cd sync_demo/ots
  • source

This last command starts CFO processes on mu2edaq04, DTC0 processes on mu2edaq05, and DTC1 processes on mu2edaq06, as well as the gateway webpage on mu2edaq06.

Open firefox on mu2edaq01: (redirected to

Follow logfiles:
  • CFO log: tail -f /home/mu2edaq/sync_demo/ots/srcs/otsdaq_mu2e/Data/Logs/
  • DTC0 log: tail -f /home/mu2edaq/sync_demo/ots/srcs/otsdaq_mu2e/Data/Logs/
  • DTC1 log: tail -f /home/mu2edaq/sync_demo/ots/srcs/otsdaq_mu2e/Data/Logs/

What happens when you click "Configure"?

  • Go to icon "State Machine"
    • click on "Parameter setup" (Lower left hand corner), select defaultSystemConfig
    • click "Initialize"
    • click "Configure"

Procedure: otsdaq instantiates the class associated with the device, then calls the ::Configure() method in the order in which the ConfigurePriority is set.

If ::configure() calls "IndicateIterationWork();", otsdaq will loop through all the devices again in the ConfigurePriority order, calling ::configure() for the devices which have indicated they still need to be touched. For the sync demo, we have the following...

  1. loop iteration 0
    • CFO: disable event start characters, disable SERDES Tx and RX, turn off event windows, turn off 40 MHz markers
    • DTC: soft reset FPGA
  2. loop iteration 1
    • CFO: set crystal/clock to 200 MHz
    • DTC: set crystal/clock to 200 MHz
  3. loop iteration 2
    • CFO:
    • DTC: configure Jitter Attenuator
  4. loop iteration 3
    • CFO: reset SERDES PLLs, reset SERDEX Tx
    • DTC:
  5. loop iteration 4
    • CFO:
    • DTC0: wait for links to lock in CFO and enabled ROCs (firmware should reset these automatically, first Tx, then Rx, upon detection of lost lock of Rx CFO in iteration 3)
    • DTC1: wait for links to lock in CFO and enabled ROCs (firmware should reset these automatically, first Tx, then Rx, upon detection of lost lock of clock Rx, above)
  6. loop iteration 5
    • CFO:
    • DTC: enable markers to all ROC links, enable Tx and Rx to all ROC links, set control register (0x9000) to 0
    • ROC: Reset DCS FIFOs (register 0), setup to align links upon receipt of characters (register 22), write/readback event window offset
  7. loop iteration 6
    • CFO: reset SERDES Rx; enable event window start characters, enable SERDES Tx and Rx, set Event Window interval time (in 200 MHz units - 5 ns), set 40 MHz marker interval time (in 40 MHz units - 25 ns)
    • All: report link status when "Configured"

What happens when you click "Start"? Perform loopback measurement

  • Click "Configure"
  • Click "Start"

Final output = average value of distributions for each ROC

The order of ROC loopback is:
  • CFOLink0, DTC0, ROC0
  • CFOLink0, DTC0, ROC1
  • ...
  • CFOLink0, DTC0, ROCN
  • CFOLink0, DTC1, ROC0
  • CFOLink0, DTC1, ROC1
  • ...
  • CFOLink0, DTCM, ROCN
  • CFOLink1, DTC1, ROC0
  • CFOLink1, DTC1, ROC1
  • ...

In the code, for each loopback step, it uses startPriority to first setup all DTCs in a chain to communicate to the specific ROC, then actually do the loopback with the CFO. Uses configuration parameter DTC:location_in_chain.

For each DTC, it either...
  • IF selecting a ROC on this DTC, write to 0x9100 -> 0x00000000. IF not doing loopback on this DTC, pass the loopback signal through: write to 0x9100 ->0x10000000
  • Select a ROC to loopback by writing to 0x9114 -> 0x00004040 | (0x101 << activeROC) [enables TX and Rx to CFO (bit 6) and appropriate ROC]
After having configured all the DTCs, on the CFO it perform the following sequence 100 times to build a histogram of delay measurements
  • Write 0x9380 -> 0x00000000 to take it out of delay measurement mode
  • Write 0x9114 -> 0x00000000 to disable tx and rx data
  • Write 0x9114 -> (0x00000101 << linkToLoopback) to enable tx and rx for the specific linkToLoopback
  • Write 0x9380 -> (0x00000100 << linkToLoopback) to put linkToLoopback into "delay measurement mode"
  • Write 0x9380 -> (0x00000101 << linkToLoopback) to begin delay measurement
  • wait 100 usec
  • Read delay value (in 200 MHz cycles) at 0x9360

What happens when you click "Stop"? Pulse CAPTAN and read timestamps of data

Configure the system, then use Macro Maker to read and write registers

  • CAPTAN (external data source)
    • 0x1001 Data count (number of pulses to send)
    • 0x1002 Data period: spacing = [(N+1) * 64] ns steps

First write 0x1002, then 0x1001 to send. Writing 0x1002 stops sending pulses.

Front-end interface software

Code to communicate with your front end is located at /home/mu2edaq/sync_demo/ots/srcs/otsdaq_mu2e/otsdaq-mu2e/FEInterfaces/

Copy and modify the CFO code for your front end...
  • CFOFrontEndInterface.h

... and add your component to CMakeLists.txt

CFO and DTC example software in: wd=/home/mu2edaq/mu2e-mrb-base/srcs/pcie_linux_kernel_module/dtcInterfaceLib/
  • CFO/DTC read/write registers in wd/
  • ROC write/read (via DTC) in wd/

To compile the code

As mu2edaq@mu2edaq01
  • cd
  • cd sync_demo/ots/
  • source
  • mrb b
To recompile after having changed package versions (e.g., if the version of pcie_linux_kernel_module changes in /home/mu2edaq/sync_demo/ots/srcs/otsdaq_mu2e/ups/product_deps)
  • mrb z
  • mrbsetenv
  • mrb b
Other commands
  • unset_all (to get rid of old dependencies)

Git commands

cd ~mu2edaq/sync_demo/ots/srcs/otsdaq_mu2e/

To compare code:
  • git status (compare present to repository)
  • git diff (compare, print out conflicts)
To merge/commit code
  • git pull (pull stuff from repository and, if possible, merge --> complain if conflicts)
  • git commit -a -m "comment" (-a = all)
  • git push (put what is committed into the repository)
Less often
  • git add (if not already in repository to track/commit a specific directory/file)
  • git stash (puts changes into a local stack)
  • git stash pop (pulls top of the stack back into directory)
  • git stash drop (gets rid of the top of the stack)

Demo setup configuration parameters

The parameters described below give the setup as follows:

One "Chain" defined by CFO link 0: CFO (mounted in mu2edaq04) -> DTC0 (in mu2edaq05) -> DTC1 (in mu2edaq06)

One or two ROCs attached to DTC0 or DTC1

One CAPTAN to send synchronous data to ROCs

Gateway application (i.e., the GUI) running on mu2edaq06 --> this means access the otsdaq top-level GUI with browser at:

Context parameters

The following "Context" parameters (software application) are easiest to look at with ActiveContext Tree-View -> XDAQContextConfiguration...

Application configuration parameters
  • Address = IP associated with device (host machine to run application)
  • Port = port associated with Address
  • Status = On/off = ienable / do not enable in software setup
device "address" "Port"
gateway ${OTS_MAIN_PORT} (i.e., 2015)
CFO0 ${OTS_MAIN_PORT}+1 (i.e., 2016)
DTC0 ${OTS_MAIN_PORT}+2 (i.e., 2017)
DTC1 ${OTS_MAIN_PORT}+3 (i.e., 2018)
CAPTAN ${OTS_MAIN_PORT} (i.e., 2019)

State machine parameters

The following state machine parameters are easiest to look at with ActiveContext Tree-View -> XDAQContextConfiguration...

Supervisor configuration parameters
  • "XXXPriority" defines order in which action is performed. Equal numbers = let the program decide.


  • ConfigurePriority = 1
  • StartPriority = 3
  • StopPriority = 4


  • ConfigurePriority = 2
  • StartPriority = 1
  • StopPriority = 2


  • ConfigurePriority = 3
  • StartPriority = 2
  • StopPriority = 3


  • ConfigurePriority = 0
  • StartPriority = 0
  • StopPriority = 1

Device parameters

The following "Device" parameters are easiest to look at under ActiveConfiguration -> TreeView -> click on double arrow next to the "Interface" Configuration -> go to Table-view


  • UID = unique identifier (e.g. CFO0)
  • ConfigureClock = 0/1 = don't/do configure the clock at "Configure"
  • DeviceIndex = [0-M] = linux filename associated with PCIExpress card (/dev/mu2eN) to communicate with card

Re: DeviceIndex, the file associated with the PCIExpress card should be set up at bootup -> see root@daq04:crontab -l, should be something like...

  • source /mu2e/ups/setup
  • setup pcie_linux_kernel_module v2_00_02 -qe15:prof


  • UID = unique identifier (e.g. DTC0)
  • ConfigureClock = 0/1 = don't/do configure the clock and jitter attenuator at "Configure"
  • LocationInChain = [0-M] = position of DTC in the chain
  • DeviceIndex = [0-M] = linux filename associated with PCIExpress card (/dev/mu2eN) to communicate with card
  • LinkToROCGroupID = unique text to associate a group of ROCs to this DTC (e.g., DTC0ROCs)
Re: DeviceIndex, the file associated with the PCIExpress card should be set up at bootup -> see root@daq04:crontab -l, should be something like...
  • source /mu2e/ups/setup
  • setup pcie_linux_kernel_module v2_00_02 -qe15:prof


  • UID = unique identifier (e.g. DTC0_ROC0)
  • Status = On/Off = talk to it or not
  • linkID = [0-6] = DTC link ID
  • EventWindowDelayOffset = time delay to start Event Window
  • ROCGroupID = exact name of ROC group in DTC device configuration (e.g., DTC0ROCs)


  • NumberOfLoopbacks = number of loopback measurements to make at "Start"
  • NumberOfCAPTANPulses = number of CAPTAN pulses to run at "Stop"

How to add stuff in otsdaq

Add a configuration group -w

  • Table editor
  • Create table --> best approach is to take a pre-existing table, modify it, then save it with a different name
Then in Configure
  • Select your current configuration group and view in group-view
  • Add member

Add a component

  • Open "Configure" icon
  • Open up the Active Context in "Tree-view"
  • Open up the small double Arrow next to "+ All Children"
  • Select "Record creation wizard"
  • Select "Front end"
  • Enter the database record name (unique): e.g., "CAPTAN0"
  • Pick the software process (XDAQ context) associated with this device: e.g., "CAPTAN0"
    • Usually each device has its own software process
  • Name a host where the software process is going to run: e.g., ""
  • Define a port where this software process is accessible: e.g., ${OTS_MAIN_PORT}+3
  • Specify an interface library (FEInterfacePluginName) to interface with the device
    • CAPTAN = FEOtsUDPTemplateInterface
    • CFO = CFOFrontEndInterface
    • DTC = DTCFrontEndInterface

Now your device should be saved in your configuration. Next step: specify the configuration parameters...

Under ActiveContext -> Tree-view -> XDAQContextConfiguration
  • Dig all the way down to LinkToFETypeConfiguration
Under Last Activated Configuration -> Tree-view
  • Select configuration type -> Table-view to see all parameters sorted by row

IF you need to add the component interface to make it available to -w

open Table Editor

Load Column Info from FEInterfaceConfiguration in Configuration Name dropdown menu

Is your component in the FEInterfacePluginName drop down menu? If not, add it