Instructions for Shifters » History » Version 85

« Previous - Version 85/90 (diff) - Next » - Current version
Nuno Barros, 03/22/2016 03:51 PM

Instructions for Shifters

This page contains information for people taking 35t shifts. See general information on shift taking before arriving at FNAL (including prerequisites such as possessing a valid Fermi ID) on this page: Shift Information.

Log into machines in ROC West

The DUNE 35t machines are located by the windows in ROC West, on the 1st floor of Wilson Hall. There are two stations, each with a keyboard, mouse and four monitors.

Log into the computers using the username dune. It will ask for your password and then principal (in this order); use your standard personal Kerberos principal and password to log in. This will require your credentials be in the .k5login file on the machine; this should be the case (it will be the responsibility of the Deputy Run Coordinator to ensure it is) but if you have problems let the DRC know (see DRC (deputy run coordinator) schedule).

If there is always someone logged in to the machine first log them out before using your Kerberos credentials to log in.
If dune is already logged in, execute the kdestroy command to delete their kerberos credentials and then a kinit -f to load yours.

Log into the ECL


In New Entry of ECL you will find the forms: Start of Shift, Shift Summary, Start of Run, End of Run Summary, etc.

Taking DAQ data

Before taking data, be sure that no one else is already using the DAQ. Check the ECL and enter a note in New Entry.

Log into the machine where the code is:

ssh -l lbnedaq

This will also require your credentials to be in the .k5login file on the machine (ask the Deputy Run Coordinator to provide this).

For instructions on running the DAQ:

For instructions on how to use the code:

TPC Specific Data Taking Instructions

This page provides instructions for how to take TPC data, specifically calibration data.

TPC Specific FEMB Configuration Check

This page provides instructions on how to check the configuration of the FEMB shaper-amplifier ASICs.

TPC Specific FEMB Configuration Setting

This page provides instructions on how to set the configuration of the FEMB shaper-amplifier ASICs.

Things to Do on Every Shift


Every 4 hours (8-9 am, noon-1pm, 4-5 pm, 8-9 pm) do the following:

If the DAQ has crashed, restart it with the following configuration and make an elog entry:
  • lbnecmd setconfig rces_and_stssps_and_ptb_calib_0.1Hz
  • lbnecmd setdaqcomps rce{00..15} penn01

Put copies of the following two plots and in the e-log (do it at all times you check the status of the DAQ):

"APA LowVoltage Current - Channel A" from

"APA Drift Voltage" from

If the run crashed, be sure that at least 10 minutes have passed since the run crashed and the plots are taken from the monitoring pages above. There is a ~10 min time delay in the plots, so appropriate time should be given for the plots to show any effect that might have caused the run to crash.

Using the Online Monitoring

Using the Event Display(s)

Taking pedestal runs

Good and Useful Runs

For offline use, it is important to know which runs ran successfully and produced useful output. Generally, there are runs which didn't crash and for which the monitoring looks sensible. Further criteria may be specified at a later time.

There exists Google documents to log these runs in. Please update it for each run which meets the criteria!

View the list of good and useful runs.
Add a new good and useful run.

Looking at data offline

Data are stored long-term in Fermilab's enstore tape system, and accessed via dCache through SAM queries. Data should be uploaded to the permanent storage system within half an hour or so of being taken, but backlog in the system or downtime or maintenance may delay that.

To access files in permanent storage, follow the instructions at .

If you need a file that's just been acquired and hasn't yet made it offline, you can copy it yourself from the online machines.
Data files for the last three days can be found on lbnedaq6, but cannot be taken directly from lbnedaq6 to a machine where you can run the offline software on them. They must be staged to gateway02 first:

$ ssh lbnedaq@lbne35t-gateway02
$ ssh lbnedaq@lbnedaq6
$ cd /storage/data
$ ltr
-rw-r--r--. 1 lbnedaq lbne  294 Nov  7 11:02 RootOutput-29a8-fa67-232c-ae28.root
-rw-r--r--. 1 lbnedaq lbne 6.4G Nov  7 11:18 RootOutput-866a-59fe-1032-6e59.root
-rw-r--r--. 1 lbnedaq lbne 170M Nov  7 11:33 lbne_r004408_sr01_20151107T172808.root  # <-- This is the file from our last successful run
-rw-r--r--. 1 lbnedaq lbne 126K Nov  7 11:33 lbne_r-_sr-_20151107T173316.root
$ exit
$ cd /data/lbnedaq/data/tmp
$ scp -p lbnedaq@lbnedaq6:/storage/data/<your file> .
$ exit
$ # Now the file is publicly accessible:
$ scp -p lbnedaq@lbne35t-gateway02:/data/lbnedaq/data/tmp/<your file> .

This gets you the file, but you will need to process it to be able to use it in the offline code. For example:

$ lar -c RunSSPToOffline.fcl <your file> -o <your processed file>

to look at the SSP data as RawOpDetWaveforms. A full larsoft tutorial is beyond the scope of this wiki. See this Tutorial to get started.

Tips to Make Life Easier

Click on the above header for a description of "known problems" (so you don't need to be unpleasantly surprised when you see them), how to identify if something's gone wrong, and some scripts which automate common actions.

DAQ Cabling

The average shifter will hopefully not have to concern themselves with this, but if so the subsequent pages will detail the relevant cabling.

Configuration Run Modes and fHiCL Documents

This page describes the major run modes of the DAQ and their configuration fHiCL documents with important parameters. Shifters will not usually have to edit these parameters directly.

DAQ components

All you need to know about the Penn Trigger Board

Useful links

Online Monitoring
Online Event Display
Nearline Monitoring
Ganglia Complete plots

DRC (deputy run coordinator) schedule

DUNE 35t Acronyms

Communication with the shifter

In addition to calling ROC-West, you can interact with the shifter and people using the daq on (or in gmail). Send a contact request to and you can chat with the shifter directly or be invited to the 35t Shift Communication group chat.

Useful phone numbers

ROC-West -- x5449
PC4 -- x3600
DAB test area -- x4716
DRC pager -- 630-255-0145 (need to dial '9 1' on a lab phone to get an outside line. After the beeps enter the number for the call to be returned on (the extension is fine!))