Midas is a DAQ system that is expected to be used for g-2. It has been used for all tracker test beams so far.

This is an overview to using MIDAS for the g-2 tracker setup. Please feel free to add to / correct this.

Content added by Tom Stuttard, Gleb Lukicov.

Midas overview

Midas is a system for collecting data from several sources and organising it into distinct events. In general, multiple software frontends representing different sources of data (e.g. detectors, slow control data etc) collect data from hardware. An event builder groups these data fragments into a data event, which can be any distinct period of data acquisition. The resulting events are written to a midas file, which can be converted to other formats including ROOT files by analyzers.

Midas provides a HTML interface for controlling data runs.

The operating parameters of Midas are stored in the online database (ODB). ODB variables can be added, edited and removed either via the HTML interface, or using the command line tool odbedit.

The specific implementation of a DAQ system using MIDAS is called a MIDAS experiment.

g-2 Tracker DAQ MIDAS overview

  • Core MIDAS installation comes from gm2midas/midas ($MIDASSYS)
  • Common frontends comes from gm2daq ($GM2DAQ_DIR)
  • Tracker frontend located in gm2-tracker-readout-daq ($GM2_TRACKER_DAQ_SRC/midas)
  • Tracker MIDAS experiment located in gm2-tracker-readout-daq ($MIDAS_ONLINE)

g-2 Tracker DAQ MIDAS experiment

Our system comprises:

  • Tracker DAQ frontend
    • $GM2_TRACKER_DAQ_SRC/midas/StrawTrackerDAQ_frontend.cpp
    • Main interface to tracker DAQ
  • Master trigger
    • $GM2DAQ_DIR/frontends/masterTriggerTracker
    • Client for TCP/IP trigger server
    • Distributes triggers from server to all other frontends
  • Trigger server
    • $GM2DAQ_DIR/frontends/masterTriggerTracker/
    • Script sending triggers to master trigger frontnend
    • Can run on raspberry pi with HW triggers attached to a pin (e.g. BOS)
    • Otherwise can send fake triggers (either from raspberry pi or locally)
  • Event builder
    • $GM2DAQ_DIR/frontends/eventBuilderNew
    • Combines data fragments from various frontends into a "per event" data stream

Our system is currently being developed for the 2015 tracker test beam at the FNAL MTest facility. Therefore it expects HW triggers corresponding to Begin Of Spill (BOS) and End Of Spill (EOS). A spill is 4s long once every 60s.

MIDAS experiment initial setup

The following assumes that MIDAS has been installed according to this guide.

WARNING: If you have been using another MIDAS experiment with the same name on the same system, make sure to clean that one first (at least remove the /dev/shm files). Similarly, only one experiment with the same name can run on the same machine.

To set up the current tracker MIDAS experiment (which is being developed for the June 2015 test beam), following this guide:

source gm2-tracker-readout-daq/software/ #If not done already
$MIDAS_ONLINE_BIN/ hard_clean #Clean any existing experiment
odbedit -c "load $MIDAS_ONLINE_DATA/cleanODB_testBeam2015.odb" #Load clean ODB
$MIDAS_ONLINE_BIN/ run_reset #Reset MIDAS run number after load
$MIDAS_ONLINE_BIN/ ALL #Init all equipment according to your system (file paths, etc)
$MIDAS_ONLINE_BIN/ start #Start MIDAS experiment
firefox $MIDAS_HOST:$MHTTP_PORT & #Open experiment web GUI

During the odbedit load command you may get messages like this:

"/Logger/Data dir = /home/tstuttard/g-2/gm2-tracker-readout-daq/midas/online/data" in file "/home/me/gm2-tracker-readout-daq/midas/online/data/cleanODB_testBeam2015.odb" does not exist locally,                                                           
set anyhow? (y/[n]):

Answer "y" to any of these, as these paths are overwritten in the script anyway.

There are a number of error messages that can be ignored during this process, see section below.

Ignorable error messages

There are a number of error messages that appear during this first set up that can be ignored (they result from sending reset commands to a system that hasn't been started for the first time yet). Examples below:

odbedit: src/system.c:233: check_shm_host: Assertion `fp != ((void *)0)' failed.   

[ODBEdit,ERROR] [system.c:1021:ss_shm_delete,ERROR] shm_unlink(/gm2StrawTracker_SYSTEM_SHM) errno 2 (No such file or directory)

odbedit: src/system.c:233: check_shm_host: Assertion `fp != ((void *)0)' failed.   

/home/me/gm2-tracker-readout-daq/midas/online/bin/ line 176: 10111 Aborted                 odbedit -c "stop run" 

/home/me/gm2-tracker-readout-daq/midas/online/bin/ //home/me//gm2midas/midas/utils/mcleanup: /bin/csh: bad interpreter: No such file or directory      

mv: cannot stat `/home/me/gm2-tracker-readout-daq/midas/online/shm/.*.SHM': No such file or directory           

mv: cannot stat `/home/me/gm2-tracker-readout-daq/midas/online/shm/.*.TXT': No such file or directory    

rm: cannot remove `/dev/shm/gm2StrawTracker*_SHM': No such file or directory                                                  

[ODBEdit,ERROR] [midas.c:3818:cm_transition,ERROR] aborting on attempt to use invalid run number 0

Notes on experiment setup

MIDAS experiment scripts

A number of scripts for helping with the MIDAS experiment are provided in $MIDAS_ONLINE_BIN:

    • Contains predefined commands (run it without args to see them)
    • Commands to start, stop etc MIDAS, also to perform various clean-up/reset tasks if Midas does a Midas
    • hard_reset is particular useful if something goes weird with the SHM files or the ODB
    • Commands for starting/stopping the trigger server
    • Pass this an equipment name as an argument and it will re-configure the clean settings in the ODB for that equipment, or use "ALL" for all equipments.

Changing MIDAS user

The file $MIDAS_ONLINE/exptab contains a hard-coded username that Midas will use to run programs. This is set according to the MIDAS_USER variable (set in If you have changed user, re-generate a new exptab file (as the new user):

$MIDAS_ONLINE_BIN/ change_user

We recommend using a dedicated user for Midas activities to remove the need to keep changing this for different users.

Running MIDAS experiment

Once the setup steps above are complete, you can start your frontends and the trigger server and begin a MIDAS run. Once the MIDAS HTML interface is running, it should stay running and not need to be restarted unless there is an error.

To start any frontends, select the programs tab in the web interface and click the "start" button next to the desired frontend. They should go green if it has worked. Their console output is redirected to a screen with the same name (to list available screens, use "screen -ls").

Example of running MIDAS

Perform the following steps to start a MIDAS run:

1) Set up environment

source gm2-tracker-readout-daq/software/ #If not done already

2) Start MIDAS

$MIDAS_ONLINE_BIN/ start #Start MIDAS experiment (if not done already)
firefox $MIDAS_HOST:$MHTTP_PORT #Open experiment web GUI (if not done already)

3) Start trigger server

Two options here, see here for raspberry pi triggering or here for fake triggering.

The simplest case is fake triggering, e.g.

$MIDAS_ONLINE_BIN/ start_fake_trigger_server

This will start the process in a screen session called trigger_server. To exit the screen, use ctrl+a, then d.

4) Start event builder frontend

  1. Click Programs on the MIDAS web interface (in firefox)
  2. Click Start Ebuilder on the event builder program and check it goes green (unless it is already started)
  3. Connect to the Ebuilder screen to check for any errors from the terminal using screen -r Ebuilder
  4. Exit the screen using ctrl+a, then d

5) Start masterTrigger frontend

  1. Click Programs on the MIDAS web interface (in firefox)
  2. Click Start masterTrigger on the event builder program and check it goes green (unless it is already started)
  3. Connect to the masterTrigger screen to check for any errors from the terminal using screen -r masterTrigger
  4. In particular check that the connection to the trigger server has been established ("Connection to trigger server (port 90000) established" or similar) - can check this in the "Messages" page of the web browser too
  5. Exit the screen using ctrl+a, then d

6) Start straw tracker frontend

  1. Click Programs on the MIDAS web interface (in firefox)
  2. Click Start StrawTrackerDAQ on the event builder program and check it goes green (unless it is already started)
  3. Connect to the StrawTrackerDAQ screen to check for any errors from the terminal using screen -r StrawTrackerDAQ
  4. Exit the screen using ctrl+a, then d

7) Start a run

Click the "Start" button in MIDAS and then "Start" again (using the default run number it suggests, which is incremented from the last run).

The first run may not start on a fresh system as there may be a run number clash with the ODB you loaded (MIDAS can get confused about this). Run a second time and it should work.

Stop the run with the "Stop" button.

8) Look at data

See here for a guide on converting the output MIDAS data to a ROOT file.

Alternatively, the MIDAS file can be directly viewed using the mdump tool, e.g. for MIDAS run 00001:

mdump -f x -x $MIDAS_ONLINE_DATA/run00001.mid.gz | more

Other notes on running

If you need to change any ODB variables, use the ODB button in the top menu bar of the homepage.

To enable/disable a frontend, change its Enabled" parameter to "n" in the ODB, e.g. for StrawTrackerDAQ frontend the paramter is *Equipment/StrawTrackerDAQ/Common/Enabled.

The Silicon Telescope frontend is disabled by default as most users won't want to use it.

Trigger system description

The Trigger Server (Raspberry Pi) receives external Begin Of Spill (BOS) and End Of Spill (EOS) signals, and it commands the Master Trigger frontend (FE). It an be found in the $GM2DAQ_DIR/frontends/masterTriggerTracker directory.

The trigger server code is written in Python (see and is running on the Raspberry Pi. This way, Raspberry Pi acts as a server, with Master Trigger as a client. When an external signal is detected on the pin of Raspberry Pi a TCP message is sent to the Master Trigger via Ethernet, as shown the diagram below.

The trigger server makes use of the RPIO python libs for raspberry pi pin communication.

A more detailed guide to the trigger setup can be found in the attached document Gm2_Tracker_Midas_Trigger.pdf.

It is also possible to generate fake triggers from the trigger server, e.g. for testing when no BOS/EOS are present. In this mode, the server internally generates BOS/EOS and sends them to the masterTrigger. Running in this mode also has the advantage that the server can be run on any machine, not just a raspberry pi. Use the fake command line argument when launching the trigger server (see examples later).

Set up raspberry pi trigger system

Set up raspberry pi

The following is a quick-start guide to setting up the trigger system on the raspberry pi. A much more detailed description of these processes can be found in the attached document Gm2_Tracker_Midas_Trigger.pdf. For UCL users, this also includes specifics of the UCL setup, including how to connect to the raspberry pi etc.

NOTE: If you don't want to set up a raspberry pi, you can still use the trigger server in "fake" mode. More details later.

(1) Setup raspberry pi basic configuration

Setup the raspberry pi as desired, ensuring it is connected to the same network as the machine masterTrigger will run on, and no firewalls prevent the socket connection between the two

(2) Install RPIO packages

Install RPIO packages and dependencies using the following commands. Note that an internet connection is required for this. and dependencies using the following commands. Note that an internet connection is required for this.

sudo apt-get install python-setuptools    
sudo easy_install-2.7 -U RPIO           

Note that we are using python 2.7 here, which should be the default of the raspberry pi (although multiple versions are preinstalled). Check this if problems occur.

(3) Connect hardware signal

The hardware trigger signal (from the accelerator at a test beam, or from a pulse generator in a lab) must be connected to one of the raspberry pi pins. See Gm2_Tracker_Midas_Trigger.pdf for details of the available pins. Also note important information of resistors, voltage levels etc in that document to avoid damaging the raspberry pi.

At UCL, we chose pin 27, and connected a pulse generator will the following settings:

  • Mode = : Pulse
  • V = 3.3V (amplitude)
  • Width = 100 us
  • Period = 10 s

*Do NOT exceed 3.3V. Ensure pulse generator output is OFF when configuring.

Longer or shorter pulses might cause issues.

The pin used must be specified when running This can be done when starting it using the -b option.

Set up software

(1) Set environment variables

These should be set in your $GM2_TRACKER_DAQ_CFG/environment-<my host>.env file.

Define the raspberry pi system:

export  #Network hostname for your raspberry pi
export RPI_USER=pi  #User for your raspberry pi
export RPI_DIR=/home/$RPI_USER/trigger_server  #Directory for the trigger server for your raspberry pi

Tell the software what host and port the trigger server is running on (e.g. the raspberry pi):

export TRIGGER_SERVER_HOST=$RPI_HOST  #These are both used by masterTrigger frontend (written to ODB)

(2) Deploy to raspberry pi

The trigger server file is stored in the git repo at $GM2DAQ_DIR/frontends/masterTriggerTracker/ To copy this to the raspberry pi, use the deploy_trigger_server command in $MIDAS_ONLINE_BIN/, e.g.

$MIDAS_ONLINE_BIN/ deploy_trigger_server

(3) Start trigger server

The trigger server can be started remotely using the $MIDAS_ONLINE_BIN/ command start_trigger_server.

The start command is:

$MIDAS_ONLINE_BIN/ start_trigger_server

This will start the process remotely on the rpi, but with the output on a local screen session called trigger_server. To exit, the screen seesion, use ctrl+a, then d.

To stop the trigger server remotely, use:

$MIDAS_ONLINE_BIN/ stop_trigger_server

If you want to change the run parameters etc, modify the args in these commands in $MIDAS_ONLINE_BIN/ Run without args to see the options (there are different options to the real/fake modes).

Setup fake trigger system

Can use this instead of a real trigger system. It will generate triggers according to the spill period and duration that are passed to it as command line options. It can be run on any machine.

To start the fake server (on your local machine, and in a screen):

$MIDAS_ONLINE_BIN/ start_fake_trigger_server

This will start in a screen session called trigger_server. To exit, the screen seesion, use ctrl+a, then d.

To stop the server:

$MIDAS_ONLINE_BIN/ stop_fake_trigger_server

If you want to change the spill length etc, modify the args in these commands in $MIDAS_ONLINE_BIN/ Run without args to see the options (there are different options for the real/fake modes).


Restart MIDAS

If a restart is required of the main MIDAS service, use the following steps:

  1. source software/ (if not done already)
  2. If required, stop midas using $MIDAS_ONLINE/bin/ stop
  3. Start midas using $MIDAS_ONLINE/bin/ start

Restart trigger server

If the trigger server stops, restart it using the following...

Raspberry pi triggering

Restart trigger server:

source software/ #If not done already
$MIDAS_ONLINE/bin/ stop_trigger_server*
$MIDAS_ONLINE/bin/ start_trigger_server*

Then restart masterTrigger frontend from the Midas HTTP GUI.

Fake triggering

Restart trigger server:

source software/ #If not done already
$MIDAS_ONLINE/bin/ stop_fake_trigger_server*
$MIDAS_ONLINE/bin/ start_fake_trigger_server*

Then restart masterTrigger frontend from the Midas HTTP GUI.

Problems with SHM files or ODB

Try a hard reset using

source software/ #If not done already
$MIDAS_ONLINE/bin/ hard_reset #Cleans system, including removing SHM files, then reloads last ODB
$MIDAS_ONLINE/bin/ start #Start MIDAS again

Useful info

Midas environment

It seems to be that when Midas executes processes, e.g. starting program from the web gui, it uses the environment (e.g. environment variables etc) it was launched with (not confirmed by any documentation I've found yet though). E.g. the environment of the user who started midas (using " start") in our implementation.

This means that for example in the "Start command" ODB entries, these commands will have environment variables available as per the user who started Midas running. As must have been sourced to use, this generally means the environment variables.

However, when using a screen commands in "Start command" ODB entries, the processes in the screen will not have this environment available. The best way around this (I've found so far at least) is to execute the screen process in bash as a string using sh -c. This way the the environment variables in the command are dereferenced to a string befre sending the command, so the screen doesn't need to know the environment variables. See example below:

screen -S <screen-name> sh -c '<some sequence of commands that include env variables>'

If your executable in the screen command needs these variables itself however (e.g. for paths), you still need to source as part of the screen string command, e.g.

screen -S <screen-name> sh -c 'source ; <some process that needs env variables to run>'

See for examples.

Starting frontends over kerberised ssh

If you are running the frontends on a different machine e.g. via ssh and the ssh is kerberos authenticated then you need to make sure your kerberos credentials are correctly delegated/authenticated on the machine to which you are connecting. The simplest way to do this is to have a: ~/.ssh/config file with the following:

Host                 gm2straw7
GSSAPIAuthentication yes
User                 gm2
ForwardX11Trusted         yes
ForwardX11                yes
GSSAPIDelegateCredentials yes

The key line is : "GSSAPIDelegateCredentials yes". This assumes the frontend will run under the gm2 account and that the gm2 account has a forwardable ticket setup. In the midas program command the host should then simply be "gmstraw7" and not "" - this forces ssh to take the above configuration into account.

When MIDAS is started it is associated with the kerberos ticket of whoever typed the " start" command and since this ticket is only valid for 24 hours then if MIDAS runs for longer than this then it remains associated with an expired kerberos ticket and the ssh connections will then subsequently fail. For the moment the way around this is to log out, kdestroy, get another ticket (kinit -f), login in again as "gm2" and then restart MIDAS ! The joys of kerberos.

Passwords and Remote access to

Web access to the ODB on gm2straw6 is now password protected - the active buttons (start FE, parameter changes etc) now have a password. This is gm20615

Web access to is possible from outside FNAL by proxying your browser. First create a kerberised ssh tunnel to

# Make sure you have a valid kerberos ticket
# Create an ssh tunnel (note this will not return you back to a prompt) and the socket is active until you ctrl-c
ssh -Nn -D 1080  
# Setup your browser to use this proxy e.g. in firefox go to Preferences/settings > Advanced > Network > 
# Tick manual proxy setup and set the Socks Host to be localhost and the port to be 1080 and "Socks V5" ticked

Warning/error messages

The following is a list of known warning and error messages when performing various Midas tasks.

Fatal errors

Start up

The following type of errors occur when you start midas for the first time if not enough virtual memory is available.

[ODBEdit,ERROR] [system.c:752:ss_shm_open,ERROR] Cannot read '/home/G2Muon/gm2/gm2-tracker-readout-daq/software/../midas/online/shm/.SYSTEM.SHM', read() returned 515854336 instead of 536887356, errno 0 (Success)
[ODBEdit,ERROR] [midas.c:5134:bm_close_buffer,ERROR] invalid buffer handle 0

or also seen:

ODBEdit,ERROR] [system.c:4531:recv_tcp2,ERROR] unexpected connection closure
[ODBEdit,ERROR] [system.c:4585:ss_recv_net_command,ERROR] error receiving network command header, see messages
[ODBEdit,ERROR] [midas.c:10389:rpc_client_call,ERROR] call to "Logger" on "localhost.localdomain" RPC "rc_transition": error, ss_recv_net_command() status 411
[ODBEdit,ERROR] [midas.c:9106:rpc_client_connect,ERROR] cannot connect to host "localhost.localdomain", port 57561: connect() returned -1, errno 111 (Connection refused)

To correct this, increase the amount of virtual memory on the machine. By default, this value will be 50% of the machine's RAM and a typical midas run will use 500 Mb per front-end. So if the machine has less than ~4Gb RAM there can be problems with this default.

mount -o remount,size=4G /dev/shm

Then alter the entry in /etc/fstab to ensure the changes remain after rebooting.
tmpfs      /dev/shm        tmpfs   defaults,size=4G        0 0

Alternatively if using a VM, assign more memory to the VM to resolve.

Buffer size is 0

Sometimes when starting a frontend you may see a messgae stating that the buffer size was expected to be XXX, but was 0. This normally means that it couldn't find the buffer, rather than its size was 0. Check the SHM path in $MIDAS_EXPTAB.

Run won't start

This may occur if there is already a run file in the $MIDAS_ONLINE_DATA directory with the run number selected in the web GUI. This can happen if you for example reload an ODB (as you will be at the old run number in that ODB). Fix by either removing the run file or selecting a new run number.

Spurious experiments appearing

An error has been seen where MIDAS thinks there are multiple experiments in the same exptab file when there is only one. An example of this problem is given below. The exptab file is:

gm2StrawTrackerTom_gm2 /home/nfs/gm2/mtest/daq-devel/stuttard/gm2-tracker-readout-daq/midas/online/shm gm2

Odbedit commands (such as those used by result in this message:

Available experiments on local computer:
0 : gm2StrawTrackerTom_gm2
1 : shm

e.g. a spurious second experiment called "shm" appears

This problem seems to come from the MIDAS exptab read, or from the resulting RPC socket read MIDAS users to request names from the server, getting confused by the long strings (maybe the string buffer it uses to parse this file is too short).

Avoid this problem by using short experiment names, and possible also shorter file paths to the shm directory. The example problem above was solved by changing the experiment names from "gm2StrawTrackerTom_gm2" to "gm2StrawTom_gm2".

Timeout on receive remote computer info

Have seen issues before when master and slave frontends try to connect via RPC. The error looks like this:

[Master,ERROR] [midas.c:9193:rpc_client_connect,ERROR] timeout on receive remote computer info: 
[Master,ERROR] [Master_frontend.cpp:787:rpc_MT_bor,ERROR] Network error
[Master,ERROR] [Master_frontend.cpp:807:rpc_MT_bor,ERROR] Cannot connect to client [MWPCDAQ]

A similar message is seen by the slave.

The error seems to be a timing issue, and does not always appear and often is removed by turning on verbose mode or adding more printing to screen. I think it might be when both master and slave to to send and receive remote computer info to each other at the same time and both get stuck waiting for the response that they have missed.

It seems to be solved by forcing the master to start first in the sequence, e.g. add this line to frontend_init:

cm_set_transition_sequence( TR_START, 499);

Non-fatal errors

The following messages may be ignored (although some should still be fixed at some point).

ODB commands

Some ODB commands (or processes using them, such as many of the $MIDAS_ONLINE/bin scripts) produce error messages that are not problematic. See examples below...

[system.c:1015:ss_shm_delete,ERROR] shm_unlink(/gm2StrawsDAQ_SYSMSG_SHM)
errno 2 (No such file or directory)

This happens when ODB tries to unlink SHM files)(e.g. removes them, such as during ODB cleaning), but there aren't any. This can occur because a clean is performed on an already clean installation for example.

Fontends compilation

In function ‘eb_begin_of_run’:
warning: implicit declaration of function ‘atoi’
In function ‘eb_user’:
warning: unused variable ‘pdata’
In function ‘load_fragment’:
warning: assignment makes integer from pointer without a cast
warning: assignment makes integer from pointer without a cast
In function ‘source_scan’:
warning: unused variable ‘plrl’
masterMT/masterMT.c:743: warning: format ‘%lu’ expects type ‘long
unsigned int’, but argument 2 has type ‘int’