FD Commissioning » History » Version 55

« Previous - Version 55/107 (diff) - Next » - Current version
Luke Corwin, 06/12/2013 12:50 PM

FD Commissioning

Specific instructions for shifters involved in commissioning of the Far Detector

Included are specific instructions for performing shift duties related to Far Detector commissioning.
Updates are included regularly after evaluation of the shifter commissioning experience.

General schedule for commissioning shifts (daytime hours):

0800-1000 Evaluate overnight commissioning run using commissioning shift checklist
1000-1500 Monitor ongoing NDOS, FD runs, perform DSO scans, short test runs as requested by FD outfitters
1500-1700 Start overnight FD commissioning run with new detector elements

General procedures to be aware of on shift

Calls from Ash River outfitters

Acknowledge the message of the call and make an entry in the ECL

Example : WH control Room will get a call when a group of APDs has been installed and has been turned over for commissioning operations

Acknowledge the message and enter into the logbook the DCM designation(s)
Call Expert to have evaluation of APD environment done (Leon Mualem or Mat Muether)
Once approved follow instructions for ramping of HV and recording of DSO scans
On completion, enter into logbook status of the channels, i.e., ready for HV operation

At times one may want to put a link in one logbook (ECL) to another logbook.

First, make the entry in the first ECL
Click the bold blue header for that entry in the first ECL; this will take you to a web page for that entry.
Right click the hyperlink beside "URL:" on this page.
Select Copy Link Location
Go two the second ECL
Click New Entry
In the new entry, click the box beside "Textile formatted:" and under "Address"
Scroll down to the "Insert link:" box.
Paste the URL into the box
In the box under "Display," type whatever words you want to link to the entry in the first ECL
Click "insert" (to the right of the "Display" box). This will insert code into the "Entry text:" box
Type the remainder of your new entry around the link.
Click "Post" to post the entry into the second ECL.

Control Room Configuration

First, you will notice 5 groups of monitors in CR:

NOVA-DAQ-1: Start and Stop Run Control, Pedestal Runner
NOVA-DAQ-2: Memory Viewer, Event Display, APD Temperature Monitor, OnMon
NOVA-DAQ-3: Ganglia, Nearline
NOVA-DAQ-4: FD, NDOS cameras
NOVA-DAQ-5: DCS Monitoring and Control

On any group, web applications can be obtained by starting FireFox from the toolbar
Click on Home to get standard web pages for shift Operations
Bookmarks include :

NOvA ECL (Electronic Collaboration Logbook), Ganglia Monitoring Program, Ash River ECL, Nearline Monitoring, FD Commissioning Shift Instructions

Switching Between NDOS and FD Monitoring in VNC :

Each of the machines usually has two virtual network connections (VNCs) running on it, one for the NDOS and one for the FD. What you usually see when first looking at the screens is actually one of these VNC connection. The FD VNC has a maroon background, and the NDOS has a black background.

Hit F8 for gui - click to exit the Full Screen mode.
NDOS is usually in session 1 and FD is usually in session 3. Choose the session you want to monitor, hit F8 and switch back to the full screen mode.
If no session exists for the detector you want, click on the appropriate icon (usually at top left) to start a new session for that detector

In case of VNC trouble

If you cannot get a VNC started using the appropriate icon, the VNC server may need to be restarted. Do this on novadaq-far-master-02 by issuing the command:

For more info on the VNCs, see [[]], [[]], and NOvA-doc-8406.

Expert Notification Procedure

Options : phone, email, logbook

For Shifters :

When a problem is encountered :

  • Look up in Redmine pages
  • solve problem with instructions

Enter into Logbook


  • contact expert :
    1. Phone
    2. if no answer on phone :
      Email Expert
      Phone Run Coordinator
    3. Enter into Logbook

For Experts performing work :

When starting work :

  1. Phone Control Room (630-840-4008)
  2. Enter into Logbook

When work is complete or stopped :

  1. Phone Control Room
  2. Enter into Logbook


Desktop Icons :

Run Pedestal Commissioning (specifically for commissioning work)
Run Pedestal (reserved for normal operations after commissioning)

Displays include: Run Control, Message Facility, Trigger scalers, DAQ Resource Manager, DAQ AppMan, TDU for active runs

To start a DSO Scan :

Click on Run Pedestal Commissioning icon on Desktop (or Run Pedestal for normal data-taking after commissioning)

2 windows will appear - a terminal window with scrolling messages and the Run Pedestal gui
click "Go" to start scan

please check here for more information (old instructions for NDOS) :

To view results:
  • Click on Check DSO Result - vs Database icon on Desktop
  • Results will pop up in OnMon Viewer
  • Make a snapshot of the Pixel Enable and Threshold results and post in the logbook

If one does not wish to deal with taking screen shots, there is an alternate method of getting the plots into the logbook:

In the OnMon view, bring up the plot you wish to save.
Right click in the white area of the plot; this will bring up the TCanvas menu.
Select SaveAs
Input name into the "filename" field of the the dialog box that appears, preferably a name that is descriptive and unique and with a .png extension (for example DSOThresholds_FD__2013-05-23_09_28_14.png).
Click OK. This will save the file into the directory ~novadaq/plotTestOffline/ You can either load them into the logbook from this directory directly or move them to /tmp/ALL as follows:
Open a terminal on novadaq-far-master-02 (note this is not novadaq-far-master) The easiest way to do this is just open a fresh terminal.
cp ~novadaq/plotTestOffline/filename.png /tmp/ALL/
Then, you can post in the logbook as above.

To load Pixel Masks and Thresholds:

FD - log onto novadaq-far-master as novadaq (ssh novadaq@novadaq-far-master)
NDOS - log onto novadaq-ctrl-master as novadaq (ssh novadaq@novadaq-ctrl-master)
then setup_online
then type <date string>
[novadaq@novadaq-far-master ~]$ 2013-03-22_14_54_38

For the FD, you can also use the DAQConfigEditor (launched from ICON on control machine 1)
File-> Open
Select FarDetGlobalConfigP1 and click OK (this may take a while to load)
Click Action -> Load Thresholds and Mask
The directory data1-a/PedestalData/PedScans should already be higlighted
Expand the directory PedScans, and you will see a list of directories in the date format YYYY-MM-DD_HH_MM_SS
Select the most recent directory and click OK
For now, load the values with exception overrides. This may also take a while, but then you are done

In the case of running DSO scans for APD Checkout or for some other unusual subset of the detector:

Edit the file


Uncomment or insert all DCMs to be included in the commissioning DSO scan; follow the pattern already in the file.

Then, double click the icon "Run Pedestal Commissioning," and proceed with the steps above, but do not load the pixel masks or thresholds from a commissioning DSO scan.

Adjust DCM Voltages Using Voltage Divider Settings:

The voltage on the DCMs are set by a voltage divider that reduces the voltage from the power supply. Normally, the 425V from the supply is reduced to 349V (when warm) and 337V (when APDs are cooled). The voltage divider uses arbitrary units that are linearly related to the actual voltages. To adjust the voltage for a given DCM or set of DCMs, we adjust the voltage divider settings.

  • Start by opening the DCS Config Editor from the icon on NOvA-DAQ-1
  • Select the tab apd_high_voltages
  • Expand the "warm" drop-down menu
  • If the DCM you wish to adjust is not listed, right-click on the list and select "Add DCM." Add the DCM you want and click "OK"
  • For the DCM you wish to adjust, double-click the number in the "voltage" column and enter the new value based (93 = 300 V, 1066 = 325 V). The nominal value is 2000, which corresponds to 349V.
  • File -> Save
  • Click yes in the pop-up window
  • Action-> Generate Recipe Scripts
  • Select ok (press return) in the confirmation window.
  • To load these new settings reboot the dcms.
  • Verify the settings were properly in the CSS GUI on the single APD view.

Voltage Divider Calculations

Start a Run:

Make sure no run is going in the partition you plan to use before you start a new run.
Double-click on the icon StartRunControl on the top left of NOVA-DAQ-1 screen

automatically gets a kerberos ticket
a dialog box appears - select Partition # and click OK
Run Control gui appears on the Desktop with Discover Resources box green

Ready for starting a run :
1. Click on Discover Resources

text changes to "Rediscover Resources" and Select Resources box also turns green

2. Click on Select Resources

Resource Manager window appears on Desktop
Check to make sure the resources you want for this run are selected
if changes have been made, red notation will be present in some of the expanded selections

if this is the case, a green window appears at the bottom - Acknowledge All Deviations - click this if correct

click OK
Reserver Resources box on RC gui turns green

3. Click on Reserve Resources

DAQApplicationManager appears on Desktop, wait until Managers, DCMApplications, and Buffer Nodes boxes turn green
Trigger Scalars, Message Facility, and TDU Control Interface windows appear on Desktop

click OK in the TDUControl dialog box indicating that "FIRMWARE NOT SUPPORTED"

Release Resources and Select Configuration boxes turn green

4. click on Select Configuration

In the dialog box that appears, choose the latest configuration file by highlighting it, then click OK

If for specific reason you want to roll back to a certain configuration, click the "more" option and choose the one you want.

Note: when you do this, make a log entry noting which configuration you use in the run, e.g., 2013-05-03_20_03_35 - the time stamp
some messages appear in the Message facility (should be green (informational messages)
Prepare Configuration box on RC gui turns green

5. Click on Prepare Configuration

more green messages on Message Facility
Load Connections box on RC gui turns green

6. Click on Load Connections

more green messages on Message Facility
Make Connections box on RC gui turns green

7. Click Make Connections

more green messages on Message Facility
Load Hardware Config. box on RC gui turns green

8. CLick Load Hardware Config.

more green messages
Configure Hardware box on RC gui turns green

9. Click Configure Hardware

lots of green messages on Message Facility

if errors, yellow and/or red messages appear on Message Facility
Can see problem on DAQAppMan window
click Abort in ConfiguringHW dialog box

Load Run Config. box on RC gui turns green

10. Click Load Run Config

more green messages
Configure Run box on RC gui turns green

11. Click Configure Run

more green messages
Begin Run box on RC gui turns green

12. Before clicking Begin Run,

If starting an NDOS run:

in a terminal window of novadaq-ctrl-master, do (example with 8 dcms): -l root "dcm-3-01-{01-02},dcm-3-{02-03}-{01-03}" 'pid=` ps aux | awk ' " ' " ' /dcm[I]OC/{print$2} ' " ' " '`; kill -STOP $pid'

Click Begin Run

Select type of run in dialog box and add a comment if you like, then click OK
messages appear, Trigger Scalars start counting

then in the terminal window: -l root "dcm-3-01-{01-02},dcm-3-{02-03}-{01-03}" 'pid=` ps aux | awk ' " ' " ' /dcm[I]OC/{print$2} ' " ' " '`; kill -CONT $pid'

If starting a FD run:

It is helpful to define an environment variable with your DCM list, for example, in a terminal window of window of novadaq-far-master:


Then, do: -l root "$dcmlist" 'pid=`ps aux|awk '"'"'/dcm[I]OC/{print$2}'"'"'`; kill -STOP $pid'

Click Begin Run

Select type of run in dialog box and add a comment if you like, then click OK
messages appear, Trigger Scalars start counting

then in the terminal window: -l root "$dcmlist" 'pid=`ps aux|awk '"'"'/dcm[I]OC/{print$2}'"'"'`; kill -CONT $pid'

13. Verify that RC Start Run entry has been automatically made in ECL, if not, make an entry indicating that a new commissioning run has started.
14. On NOVA-DAQ-2, get a kerberos ticket, then start

Memory Viewer using Desktop icon (use shared memory P000 for FarDet, P001 for NDOS)

click Connect to Memory

Event Display using Desktop icon

in dialog box, select Partition # and click OK
start automatic scrolling by clicking on ">" at top of EVD

Online Monitoring (OnMon)

click OnMon Producer desktop icon

select Partition # in dialog box and click OK
verify that events are scrolling in the producer window

click OnMon Viewer to start histogram viewer

select Partition number in dialog box and click OK
OnMon window appears

select histograms to view

End a Run:

On RC gui, Pause Run and End Run boxes should be green
1. Click on End Run box on RC gui

Comment on why run ended - click OK
Release Resources, Select Configuration, Load Connections, and Begin Run boxes should turn green

2. Click Release Resources on RC gui

Trigger Scalars, TDU COntrol, Message Facility, Resource Manager windows disappear
DAQ Application Manager shows pink boxes for DCMs and Buffer Nodes, green/pink on Managers box
Rediscover Resources, Select Resources, and Reserve Resources on RC gui are all green
If this doesn't work, you will have to click "Abort" in a popup box

then click "Stop System" in DAQ App Manager Gui

3. Click Stop Run Control icon on desktop

select Partition # in dialog box and click OK
RC gui, DAQ APPManager windows disappear

4. Ready for new run


Desktop icons:

Event Display

2 icons on desktop - "ART Event Display" and "ART Event Display - STOP"

Online Monitoring (OnMon Producer, Viewer)

2 icons on desktop for Producer - "OnMon Producer" and "OnMon - STOP"
2 icons on desktop for Viewer - "OnMon Viewer" and "OnMon Viewer - STOP"

Online Monitoring Shifter Wiki

Memory Viewer

start with "Memory Viewer" icon on desktop
stop by closing memory viewer window

APD Temperature Monitor

2 icons on desktop - "APD Temp Monitor" and "APD Temp Monitor - STOP"

DSO Results, DSO vs Reference

desktop icon "Check DSO Results" displays latest DSO result through OnMon Viewer
desktop icon "Check DSO Result vs Database" displays latest DSO result and compares it to previous result also through OnMon Viewer

Web Apps:

Ganglia (bookmarks for FD and NDOS)
Very good monitor of the state of DCMs. To display DCM cpu report histograms:
  1. Choose DCM in the "Choose a Source" box
  2. Then, in the "Metric" box, choose cpu_report
  3. Then, if you want to see just the DCMs used in the current run, click on the Partition # of the run
  4. You can also choose the time included in the displays from the last hour to year!

Overview histograms are provided as well as cpu use for the DCMs in the selected Partition. For normal running, cpu use should be well under 100% for each DCM.

Nearline (bookmark)

Web Apps:

FD, NDOS Cameras

DCS Monitoring and Control of Detector Power Supplies

Commissioning Shift Checklist

Use FD Commissioning Report form in the ECL:

Select "New Entry"
In the pop-up window, select "FD Commissioning Report" in Form box
Select "Commissioning" in the Category box
Select "FD Commissioning" in the Tags box

Instructions for each form entry are included on the form

Snapshots of plots for report:

Use the snapshot icon in the toolbar to take a picture of plots
Save to folder ALL
This folder is accessible on all desktops - directory /tmp/ALL from insert picture on form

Commissioning Run Exec Summary