Project

General

Profile

How tos

This is where you'll find out how to:

Start DAQ VNC server at evb

If evb was down or by any reason the VNC server at evb is down, you can log into evb as uboonedaq and restart the server by

~/startVNC.sh

Run control scripts

Ganglia metrics

The first time you access Ganglia you will need to:
1. Go to your browser settings and tell it to make connections according to this "Proxy AutoConnect" file: https://cdcvs.fnal.gov/redmine/attachments/24625/lartf.pac. If you are having trouble, you may need to update the proxy settings on your browser. See How_to_make_ssh_tunnels_to_ubdaq_internal_webservers. You can also find other instructions at RC Guide: Viewing the ganglia monitoring page.

You should only need to set these proxy settings once. After that you can leave it set at all times. You will need to follow the rest of the instructions (to set up an ssh tunnel and connect to Ganglia) every time you want to access the metrics:

Follow the rest of these instructions every time you want to use Ganglia:
2. Use ssh to make a dynamic SOCKS proxy server on port 8881 on your machine, via ubdaq-prod-ws01:

ssh -D 8881 ubdaq-prod-ws01.fnal.gov

You should now be able to access anything inside the LArTF network (including Ganglia) from your machine.

3. You can access Ganglia here: Ganglia server
The username is uboone, and the password is the common DAQ password xxxxxxxxxxx (concatenate the MicroBooNE DocDB user name, the number two, and the first syllable in the word preceding "xxxxxxxxxxx" in this sentence).

4. To look at the ganglia metrics start by choosing a machine, for example ubdaq-prod-evb-priv, in the drop-down box that says “--Choose a Node”. See the New Expert's guide to diagnosing DAQ problems for a starting point on where to look in the ganglia metrics, or consult the list of ganglia metrics to find out what each one shows.

Extra credit: with this ssh connection you can also access the Online Monitor Lizard.

DAQ logs

  • The DAQ log files for the current run are located at
    /home/uboonedaq/daqlogs/uboonedaq/
    
  • The DAQ log files for previous runs are located at
    /home/uboonedaq/daqlogs/uboonedaq/old
    
In SEB Logs...
  • Search for “ProcessingFragmentsStart” to get to start of run
  • Should see “StopRunRequest” when receives a stop
  • See fragments getting sent!
In EVB log
  • Search for “ProcessingFragmentsStart” to get to start of run
  • See reception of fragments, insertion to EventStore
  • Print of global header on event written to disk

examples from Wes:
~uboonedaq/daqlogs/uboonedaq/old/assembler-uboonedaq-12931- 09.23.2017-21.05.28.out.log
~uboonedaq/daqlogs/uboonedaq/old/seb-9-uboonedaq-12931- 09.23.2017-21.05.28.out.log

Run configuration database

When you upload a new run configuration, don't forget to add it to the List of config files and their purposes!

DAQ development wiki

Get DAQ statistics

The information about the various reasons causing for run failure and about the run lengths is all kept in the elog, and a python script can be run to retrieve this information and plot it nicely.
Since run conditions can change, we would like to show the DAQ statistics for runs with the same config.

log into one of the online machines, for example ubdaq-prod-ws01/:

source /uboonenew/setup
setup root v5_34_23 -q e6:prof
python /home/uboonedaq/DAQelogStatistics/getDAQelogStatistics.py -b

Then you're going to have to answer some questions, about which configs you would like to run over. Supply the script with the config number, and short description of it which will be shown on the plots. Type 0 if you're done with the configs instead of a config number. Then tell it from which you would like to run the statistic test from in the requested form.
Last you will have to choose between the option 'get' and 'plot'. The first is to be ran the first time, and will actually get the information from the elog. The 'plot' option can be used if you just wish to plot some the information about a subset of the config files or modify the titles without having to wait for the information to be grabbed again.

All plots will be saved in a directory named ./plots/ created in your working area.

Run with masked SEBs

In order to run without one or more SEBs, one should prepare a config excluding the masked SEBs from the AssemblerAppevb registration section, like the following example when intended to mask SEB03:


assemblerAppevb: [  {
  dispatcher:{
    shmKey : 0x0deaf0
    shmSize : 1000000000
  }
  .
  .
  .
  }
  registration:{
    seb1:{
      name : "sebpriv01" 
      port : 33421
      timeout : 5e3
      type : "TPC" 
    }
    seb10:{
      name : "sebpriv10" 
      port : 33421
      timeout : 5e3
      type : "PMT" 
    }
    seb2:{
      name : "sebpriv02" 
      port : 33421
      timeout : 5e3
      type : "TPC" 
    }
    seb4:{
      name : "sebpriv04" 
      port : 33421
     timeout : 5e3
      type : "TPC" 
    }
    seb5:{
      name : "sebpriv05" 
      port : 33421
      timeout : 5e3
      type : "TPC" 
    }
    seb6:{
      name : "sebpriv06" 
      port : 33421
      timeout : 5e3
      type : "TPC" 
    }
    seb7:{
      name : "sebpriv07" 
      port : 33421
      timeout : 5e3
      type : "TPC" 
    }
    seb8:{
      name : "sebpriv08" 
      port : 33421
      timeout : 5e3
      type : "TPC" 
    }
    seb9:{
      name : "sebpriv09" 
      port : 33421
      timeout : 5e3
      type : "TPC" 
    }
    trg:{
      name : "sebpriv10" 
      port : 33422
      timeout : 5e3
      type : "TRIGGER" 
    }
  }

Then, when running runConsoleDAQ use the option

-- seb-mask 
and the decimal value of a list of 1/0 bits setting the SEBs status from right to left.
For example, if you wish to mask SEB01 the list of bits will be 1111111110 and the decimal value of it is 1022. if you wish to mask SEB03 the list of bits will be 1111111011 and the decimal value of it is 1019.

How to log in to the DAQ crates

To look at the online machines in the DAQ room use the following original username and password: 00000000 00000000.

How to restart ganglia when not working

Inside evb, inside root mode type:

service gmond restart
service gmetad restart