How To... What to do while on shift DAQ Trouble Shooting Guide

Checking DCM Synchronization

Important: click here for instructions to follow while we are using 2 Master TDUs

Updated 2012-03-21


Most of the material on this page is for reference. If things have been set-up correctly, the highlighted terms in this overview are sufficient to check the synchronization:

  • The DCMs need to have their 64 MHz clocks in synch in order to get good data.
  • At the start of the run, TDUControl issues a synch command to make this happen.
  • To check this, the TDUControl automatically issues periodic (currently every 30 sec.) "timing pings" to provide the DCMs with a simultaneous signal (modulo propagation delays) to timestamp.
  • The DCMs all report their 64 MHz timestamp of the timing ping to the (log) Message Facility.
  • The shifter uses the script, which mines the log file and reports whether all DCMs are reporting the same time.
  • If they are not in synch, the shifter needs to promptly issue an "atomic synch" from TDUControl.

TDUControl Automatic Pinging

  • TDUControl issues periodic timing pings (not synchs) during a run.
  • If the run is not yet going, but Run Control is issuing heartbeats, you'll likely see that no pings are being issued as indicated by the "LEDs". (That's OK when a run is not going.)

  • If the "Heart Beats LED" is red, then either the the system is not yet active, or Run Control is in a bad state.
  • Once a run is going and TDUControl is receiving Run Control heartbeats the respective "LEDs" should be green:

  • At this point, it's time to look at the output of to see if the DCMs are well synched.

Return to Overview.

DCM sync query script

This script checks the log files of the Message Viewer to evaluate the results of the latest Timing Ping. This will be replaced (soon) by the ErrorHandler/MessageAnalyzer.

Have a terminal window where you can run the script

This script can be run in any terminal window, but it's best to have a dedicated one so it can be left running in a loop.
There may already be such a window, showing output such as

All DCMs in synch within 0 ticks/ ( Limit = 10)

(Latest ping=2011-Dec-23 18:23:26.558094562500 UTC)
or, see the Sample Output section below.

If there is no such window, do the following:
  1. Log into novadaq-ctrl-master:
  2. Set up the NOVA-DAQ Environment:
    •  [novadaq@novadaq-ctrl-master ~]$ setup_online

Running the script

  • To leave running in a loop, copy and paste the following command into the terminal window mentioned above:
     while [ 1 ]; do; sleep 15; done 
  • To run without looping, just type 

Sample Output

If all the DCMs are in sync the script will return: (All is OK in this case.)

All DCMs in synch within 0 ticks/ ( Limit = 10)

(Latest ping=2011-Dec-23 19:11:56.558285812500 UTC)

Here is a sample of what it looks like if some DCMs are not in synch:

If the latest pings are in synch but too old to use you'll see

Error - Most recent ping is too old (945 sec) to use:
Latest ping=2011-Dec-23 18:20:56.557463937500 UTC

 Please issue ping and run again.
This is OK if the run is not going.
If there have been no pings since the run started, you will see
No timing pings found since run start!
Please issue a timing ping and try again.
This should not happen if the TDUControl is operating properly.

Return to Overview.

What to do with the results

  • If all the DCMs are synched, no action is required.
  • If one or more DCMs are out of sync issue an Atomic Sync
    1. From the Config Menu on the TDU Control Interface, select Enable Expert Mode
    2. In the now-visible Timing Control tab click on the Atomic Sync button
    3. A new ping will be sent within 30 seconds.
    4. (or you can force one from the "Timing" menu in TDUControl)
    5. Check the script output (either run it again, or wait a few seconds if it's looping).
    6. If DCMs are still out of sync after several attempts, call an expert
  • If the latest ping is too old or missing, and the run is in progress, issue one manually:

    **If this problem persists, call an expert.
Return to Overview.
How To... What to do while on shift DAQ Trouble Shooting Guide