Project

General

Profile

FD Commissioning

This information is outdated. Please go here for in date information (L. Suter 27/03/15)
https://cdcvs.fnal.gov/redmine/projects/novaoperations/wiki/ShiftBulletinBoard

General Shift Plans and Instructions

When you arrive to start your shift, if no shift immediately precedes yours, you might have to unlock the 12th floor Control Room - the key is held in a magnetic clip on the wall underneath the cabinets in Andrew Norman's office on the 12th floor crossover. Please remember to return the key to this location when you are done with it.

For shift instructions and procedures, refer to this document online and/or use the "Shift Handbook", a white binder in the 12th floor Control Room.

General instructions for all shifts

Start Shift form : At beginning of your shift, fill out "Start Shift" form in the ECL, following the instructions in the form.

login to the ECL using your account, click on the "New Entry" item in the left column menu, select the "Start Shift" form from the "Forms" drop-down menu.
also, if runs are active, use the green "Change" button on the RC gui to change to your name.

Read the "Daily Run Plan" entry in the ECL (normally posted ~7:30AM daily) for specific instructions of the day.

Beam checklist : At least once during the shift, fill out the "Beam" checklist from the drop-down menu in the "Forms" box in "New Entry", following the instructions in the form.

Control Room Checklist : At least once during the shift, fill out the "Control Room Checklist" form, following all instructions on the form.

End Shift : At the end of your shift, fill out the "End Shift" form.

Additional Shift forms and checklists

Stable Run Checklist : During your shift, whenever you start a run, after the run is started and the monitoring programs have been running for ~10 minutes, fill out the "Stable Run Checklist" form. Filling out this checklist will ensure that all necessary monitoring programs used in the commissioning process and that data quality monitoring programs are operational.

DSO Scan Report : If you do a DSO Scan (run Pedestal Runner from DAQ-1 desktop icon), for each one (except those associated with the "New APD Checkout" form) use the "DSO Scan Report" form in the ECL, following the instructions on the form.

For commissioning shifts, several other forms are necessary :
  • FEB Checkout form - used to monitor FEBs installed on DCMs before APDs are installed.
  • New APD Checkout form - used to ramp HV on DCMs with new APD installations. This is normally done in the early afternoon - a DCMs worth of APDs (64) can be installed in a half day starting at 7AM, so will normally be ready for checkout after lunch.
  • FD Commissioning Report form - used to evaluate new detector installations. This is normally done after an overnight or weekend run of a new APDs. Follow instructions on the form and include as much information as you can about the state of the new detector configuration, including snapshots of relevant plots and monitoring program conditions.

Duties and schedule for shifters involved in commissioning of the Far Detector

Typical schedule for shifts with commissioning activity (day/swing hours):

0800-0830 Fill out Start Shift form, check e-logs and Daily Run Plan.
0830-1000 Evaluate overnight commissioning run using FD Commissioning Report form
1000-1400 Fill out Beam checklist, Control Room checklist, monitor ongoing NDOS, FD runs, perform DSO scans, assist experts with test runs if requested
1400-1600 Fill out New APD Checkout form for new APD installation if requested by Ash River outfitters
1600-1700 Do final DSO Scan of new detector elements filling out DSO Scan Report, start overnight FD commissioning run
1700-2300 Fill out Stable Run Checklist, Beam checklist, Control Room checklist, monitor ongoing NDOS andFD runs
2300-2400 Make sure current runs are active and all monitoring programs are running normally, fill out End Shift form

General procedures to be aware of on shift

Issuing of underground access keys

Online processes to check badge IDs (FLUFFY) and permit numbers are available on web browser on one of the DAQ-4 screens
When a request is made for underground access, type in the badge number to obtain training status (if green, OK to issue key - if red, cannot issue key)
User should know which permit number they will use.
Users sign log book next to door and get key for access.

Calls from Ash River outfitters

Acknowledge the message of the call and make an entry in the ECL

Example : WH control Room will get a call when a group of APDs has been installed and has been turned over for commissioning operations

Acknowledge the message and enter into the logbook the DCM designation(s)
Check IFIX monitor Dry Air status and note entries in the New APD Checkout form

If pressure and dew point values are not within safe range, wait until they reach safe values to continue - if they don't improve, note in ECL

Once dew point and pressure are OK, continue with New APD Checkout, following instructions for ramping of HV and recording of DSO scans
On completion, enter into logbook status of the channels, i.e., ready for HV operation

Linking entries between NOvA ECL and NOvA Ash River ECL

First, make the entry in the first ECL
Click the bold blue header for that entry in the first ECL; this will take you to a web page for that entry.
Right click the hyperlink beside "URL:" on this page.
Select Copy Link Location
Go to the second ECL
Click New Entry
In the new entry, click the box beside "Textile formatted:" and under "Address"
Scroll down to the "Insert link:" box.
Paste the URL into the box
In the box under "Display," type whatever words you want to link to the entry in the first ECL
Click "insert" (to the right of the "Display" box). This will insert code into the "Entry text:" box
Type the remainder of your new entry around the link.
Click "Post" to post the entry into the second ECL.

Wilson Hall Control Room Configuration

First, you will notice 6 groups of monitors in CR:

NOVA-DAQ-1 (group of 4 monitors): Start and Stop Run Control, Pedestal Runner
NOVA-DAQ-2 (group of 4 monitors): Memory Viewer, Event Display, APD Temperature Monitor, OnMon
NOVA-DAQ-3 (group of 4 monitors): Ganglia, Nearline
NOVA-DAQ-4 (2 monitors above DAQ-2, -3, 2 monitors stacked on table): ND Cameras / FD cameras / Beam displays / Underground access pages (FLUFFY)
NOVA-DAQ-5 (4 monitors in a row on table): DCS Monitoring and Control
iFIX machine for the Gas and Water systems (single screen on table).

On any group, web applications can be obtained by starting FireFox from the toolbar
In Firefox browser, click on Home to get standard web pages for Shift Operations
Bookmarks include :

NOvA ECL (Electronic Collaboration Logbook), FarDet Ganglia Monitoring Program, NDOS Ganglia, NOvA Ash River ECL, Nearline Monitoring, FD Commissioning Shift Instructions

Switching Between NDOS and FD Monitoring in VNC :

Each of the machines usually has two virtual network connections (VNCs) running on it, one for the NDOS and one for the FD. With the current setup, there is room for 2 more sessions per machine). What you usually see when first looking at the screens is one of the VNC sessions. The FD VNC has a maroon background, and the NDOS has a black background.

To switch between VNC sessions:
Click F8 for gui - click to uncheck the Full Screen mode.
NDOS is usually in session 1 and FD is usually in session 3. Choose the session you want to monitor, hit F8 and switch back to the full screen mode.
If no session exists for the detector you want, click on the appropriate icon (usually at top left) to start a new session for that detector

In case of VNC trouble

If you cannot get a VNC started using the appropriate icon, the VNC server may need to be restarted. Do this on novadaq-far-master-02 by issuing the command:

start-vncservers.sh

For more info on the VNCs, see [[https://cdcvs.fnal.gov/redmine/projects/novadaqcontrolroomsw/wiki/Conventions_and_Tools]], [[https://cdcvs.fnal.gov/redmine/projects/novadaqcontrolroomsw/wiki/VNCSetup]], and NOvA-doc-8406.

Expert Notification Procedure

Options : phone, email, logbook

For Shifters :

When a problem is encountered :

  • Look up in Redmine pages
  • solve problem with instructions

Enter into Logbook

or

  • contact expert :
    1. Phone
    2. if no answer on phone :
      Email Expert
      Phone Run Coordinator
    3. Enter into Logbook

For Experts performing work :

When starting work :

  1. Phone Control Room (630-840-4008)
  2. Enter into Logbook

When work is complete or stopped :

  1. Phone Control Room
  2. Enter into Logbook

On NOVA-DAQ-1

Desktop Icons :

StartRunControl
StopRunControl
Run Pedestal Commissioning (specifically for commissioning work)
Run Pedestal (reserved for normal operations after commissioning)

Displays include: Run Control, Message Facility, Trigger scalers, DAQ Resource Manager, DAQ AppMan, TDU for active runs

To run a DSO Scan :

Before beginning a DSO Scan, follow the "Ending a Run" procedure (ensure that resources you need for the DSO scan, i.e., DCMs, are not being used in a run).
There are 3 icons on the FarDet DAQ-1 screen - "Run Pedestal" (normally used for large parts of the detector) and "Run Pedestal Commissioning" (normally used for single DCMs during New APD Checkout) that control the DSO scans which define masked channels and thresholds for the detector.

Be sure that all DCMs which are included in the DSO scan are readable. Check on the CSS Gui window (APD Temp Monitor) each DCM on the DiBlock page (not only Overall view) are in (light/dark) green color, not white ("FEB inactive") or pink!

Before running a DSO scan, check the appropriate xml file for the icon you will click and edit if necessary :
  • for "Run Pedestal" - edit /nova/config/FarDet/pedestals/PedestalConfiguration_Standard_FarDet.xml
  • for "Run Pedestal Commissioning" - edit /nova/config/FarDet/pedestals/PedestalConfiguration_Standard_FarDet_Commissioning.xml
  • for "APDless Run Pedestal" - edit /nova/config/FarDet/pedestals/PedestalConfiguration_NoAPD_FarDet.xml

by uncommenting or inserting DCMs to be included in the DSO scan following the pattern already present in the file.

To start the scanning process:

Double-click on the "Run Pedestal" or "Run Pedestal Commissioning" desktop icon on the FarDet DAQ-1 screen
  • 2 windows will appear - a terminal window with scrolling messages and the Run Pedestal gui
  • On the gui, click "Go" to start scan

On each DCM line, the percentage will increase, coloring the line blue until the scan is complete.
When the scan is complete, at the top of the gui, the word "DONE" will appear. (Ignore other messages meant for experts, e.g., "Broken", or "Normal Speed", etc.)

To view the results of the scan:

  • Double-click on Check DSO Result - vs Database icon on FarDet DAQ-2 screen
  • In the pop-up box which appears, select the partition you will use these threshold and mask values in (typically 1)
  • Results will pop up in an OnMon Viewer
  • In ECL New Entry, select the "DSO Scan Report" item from the Forms box at the top
  • Follow the instruction for filling out this report and including pictures from the results (PixelEnableMask, Thresholds, and differences from the reference for each)
If one does not wish to deal with taking screen shots, there is an alternate method of getting the plots into the logbook:
  • In the OnMon view, bring up the plot you wish to save.
  • Right click in the white area of the plot; this will bring up the TCanvas menu.
  • Select SaveAs
  • Input name into the "filename" field of the the dialog box that appears, preferably a name that is descriptive and unique and with a .png extension (for example DSOThresholds_FD__2013-05-23_09_28_14.png).
  • Click OK. This will save the file into the directory ~novadaq/plotTestOffline/ You can either load them into the logbook from this directory directly or move them to /tmp/ALL as follows:
  • Open a terminal on novadaq-far-master-02 (note this is not novadaq-far-master) The easiest way to do this is just open a fresh terminal.
  • cp ~novadaq/plotTestOffline/filename.png /tmp/ALL/
  • Then, you can post in the logbook as above.

To load Pixel Masks and Thresholds:

  • Double-click the DAQConfigEditor desktop icon on the FarDet DAQ-1 screen
  • Click File-> Open
  • Select FarDetGlobalConfigP1 (for partition 1 running) and click OK (this may take a while to load - wait until you see the pop-up box complete and disappear)
  • Click Action -> Load Thresholds and Mask
  • The directory data1-a/PedestalData/PedScans should already be higlighted
  • Expand the directory PedScans, and you will see a list of directories in the date format YYYY-MM-DD_HH_MM_SS
  • Select the most recent directory (or the one you want to use if so directed) and click OK
  • Respond to the pop-up boxes, select "Append" in the final box unless specifically asked to do differently.
  • This may take some time to complete - wait until you see the pop-up box complete and disappear.
  • Finish the "DSO Scan Report" and post it to the ECL.

In the case of running DSO scans for "New APD Checkout":

  • edit /nova/config/FarDet/pedestals/PedestalConfiguration_Standard_FarDet_Commissioning.xml

Uncomment or insert all DCMs to be included in the commissioning DSO scan; follow the pattern already in the file.

  • Double-click the icon "Run Pedestal Commissioning," and proceed with the steps above, but do NOT load the pixel masks or thresholds from the scan.

Starting a Run:

  • Make sure no run is going in the partition you plan to use before you start a new run.
  • Check to see if the Resource Manager is running (a window should be present on the DAQ-1 screen)

if not, start the Resource Manager using the desktop icon "Resource Manager Start"

  • Double-click on the icon "StartRunControl" on the left of NOVA-DAQ-1 screen

automatically gets a kerberos ticket
a dialog box appears - select Partition # and click "OK"
"Run Control" gui appears on the Desktop with "Discover Resources" box green

Ready for starting a run :
1. Click "Discover Resources"

text changes to "Rediscover Resources" and "Select Resources" box also turns green

2. Click "Select Resources"

"Resource Manager" window appears on Desktop
Check to make sure the resources you want for this run are selected
if changes have been made, red notation will be present in some of the expanded selections

if this is the case, a green window appears at the bottom - "Acknowledge All Deviations" - click this if correct

click "OK"
"Reserver Resources" box on RC gui turns green

3. Click "Reserve Resources"

"DAQApplicationManager" appears on Desktop, wait until Managers, DCMApplications, and Buffer Nodes boxes turn green
Trigger Scalars, Message Facility, and TDU Control Interface windows appear on Desktop

for NDOS only, click OK in the TDUControl dialog box indicating that "FIRMWARE NOT SUPPORTED"

"Release Resources" and "Select Configuration" boxes turn green

4. click "Select Configuration"

In the dialog box that appears, choose the latest configuration file by highlighting the bottom line, for NDOS, the configuration is "InitialGlobalConfiguration", for FarDet, use "FarDetGlobalConfigP1"

If for some reason you want to use an older configuration, click the "more" option and choose the one you want.

Click "OK"
some messages appear in the "Message Facility" (should be green (informational messages)
"Prepare Configuration" box on RC gui turns green

5. Click "Prepare Configuration"

more green messages on "Message Facility"
"Load Connections" box on RC gui turns green

6. Click "Load Connections"

more green messages on "Message Facility"
"Make Connections" box on RC gui turns green

7. Click "Make Connections"

more green messages on "Message Facility"
"Load Hardware Configuration" box on RC gui turns green

8. Click "Load Hardware Configuration"

more green messages
"Configure Hardware" box on RC gui turns green

9. Click "Configure Hardware"

lots of green messages on "Message Facility"

if errors, yellow and/or red messages appear on "Message Facility"
Can see problem on DAQApplicationManager window (yellow, red, or pink colors)
click Abort in "ConfiguringHW" dialog box

"Load Run Configuration" box on RC gui turns green

10. Click "Load Run Configuration"

more green messages
"Configure Run" box on RC gui turns green

11. Click "Configure Run"

more green messages
"Begin Run" box on RC gui turns green

12. Before clicking Begin Run,

If starting an NDOS run:

Click "Begin Run"
Select type of run in dialog box and add a comment if you like, then click "OK"
messages appear, Trigger Scalars start counting

If starting a FarDet run:

It is helpful to define an environment variable with your DCM list, for example, in a terminal window of window of novadaq-far-master:

 dcmlist="dcm-2-02-0{1-6},dcm-2-03-{01-12}" 

Then, do:

 rgang.py -l root "$dcmlist" 'pid=`ps aux|awk '"'"'/dcm[I]OC/{print$2}'"'"'`; kill -STOP $pid'

Click "Begin Run"
Select type of run in dialog box and add a comment if you like, then click "OK"
messages appear, Trigger Scalars start counting

then in the terminal window:

  rgang.py -l root "$dcmlist" 'pid=`ps aux|awk '"'"'/dcm[I]OC/{print$2}'"'"'`; kill -CONT $pid'

13. After a few minutes, verify that "RC Start Run" entry has been automatically made in ECL, if not, make an entry indicating that a new commissioning run has started.

14. On NOVA-DAQ-2, get a kerberos ticket, then :

Start "Memory Viewer" using Desktop icon (use shared memory P000 for Partition 0 on FarDet, P001 for Partition 1 on FarDet, P001 for NDOS)

click "Connect to Memory"

"APD Temperature Monitor" should ALWAYS be active, if not, restart using desktop icon
Start "Event Display" using Desktop icon

in dialog box, select Partition # and click "OK"
start automatic scrolling by clicking on ">" at top of EVD

Start Online Monitoring (OnMon)

click "OnMon Producer" desktop icon

select Partition # in dialog box and click "OK"
verify that events are scrolling in the producer window

click "OnMon Viewer" to start plot viewer

select Partition # in dialog box and click "OK"
OnMon window appears

select plots to view

15. On NOVA-DAQ-3, open a browser window (FireFox icon on toolbar)

Click the "Ganglia Cluster" tab to view web monitor

select "DCM" in the mode box
select Partition # from choices
select "cpu_report from box to monitor DCM cpu
select desired time period from box

Click the "Nearline Monitor" tab to view web monitor

Ending a Run:

On RC gui, Pause Run and End Run boxes should be green
1. Fill out the ECL End Run form.

Click on End Run box on RC gui.
Comment on why run ended - click OK
Release Resources, Select Configuration, Load Connections, and Begin Run boxes should turn green

2. Click Release Resources on RC gui

Trigger Scalars, TDU COntrol, Message Facility, Resource Manager windows disappear
DAQ Application Manager shows pink boxes for DCMs and Buffer Nodes, green/pink on Managers box
Rediscover Resources, Select Resources, and Reserve Resources on RC gui are all green
If this doesn't work, you will have to click "Abort" in a popup box

then click "Stop System" in DAQ App Manager Gui

3. Click Stop Run Control icon on desktop

select Partition # in dialog box and click OK
RC gui, DAQ APPManager windows disappear

4. Ready for new run

On NOvA-DAQ-2

Desktop icons:

  • Event Display

2 icons on desktop - "ART Event Display" and "ART Event Display - STOP"

  • Online Monitoring (OnMon Producer, Viewer)

2 icons on desktop for Producer - "OnMon Producer" and "OnMon - STOP"
2 icons on desktop for Viewer - "OnMon Viewer" and "OnMon Viewer - STOP"

  • Memory Viewer

start with "Memory Viewer" icon on desktop
stop by closing memory viewer window

  • APD Temperature Monitor

2 icons on desktop - "APD Temp Monitor" and "APD Temp Monitor - STOP"

If color on FEBs is brown, the cooling inhibit bit needs to be toggled. This can be accomplished with the following procedure :

In a terminal window:

ssh novadcs@novadcs-far-master-02
setup_novadcs
ps aux|grep regIOC (CHECK IF THERE IS A regIOC PROCESS)
(if no regIOC process) startRegIOC.sh -d 2
caput det-2:_da_flow_status_0 0 (this disables the cooling inhibit)

should be fixed now

  • DSO Results, DSO vs Reference

desktop icon "Check DSO Results" displays latest DSO result through OnMon Viewer
desktop icon "Check DSO Result vs Database" displays latest DSO result and compares it to previous result also through OnMon Viewer

Adjust DCM Voltages Using Voltage Divider Settings (normally done with an expert present):

The voltages on the DCMs are set by a voltage divider that reduces the voltage from the power supply. At present, the 425V from the supply is reduced to 349V (when warm) and 337V (when APDs are cooled). The voltage divider uses units that are linearly related to the actual voltages. To adjust the voltage for a given DCM or set of DCMs, we adjust the voltage divider settings.

  • Start by opening the DCS Config Editor from the icon on NOvA-DAQ-2
  • Select the tab apd_high_voltages
  • Expand the "warm" drop-down menu
  • If the DCM you wish to adjust is not listed, right-click on the list and select "Add DCM." Add the DCM you want and click "OK"
  • For the DCM you wish to adjust, double-click the number in the "voltage" column and enter the new value based (93 = 300 V, 1066 = 325 V). The nominal value is 2000, which corresponds to 349V.
  • File -> Save
  • Click yes in the pop-up window
  • Action-> Generate Recipe Scripts
  • Select ok (press return) in the confirmation window to load these new settings reboot the dcms.
  • Verify the settings were properly recorded in the APD Temperature Monitor (CSS GUI) on the single APD view.

Voltage Divider Calculations

How to Unzoom the Event Display

  • pause the EVD
  • go to the edit menu and choose: configure drawing->Geometry Drawing Options
  • at the bottom next to Zoom, choose the radio button "full"
  • hit "Apply" at the very bottom

On NOvA-DAQ-3
Web Apps:

Ganglia (bookmark)

  • Very good monitor of the state of DCMs. To display DCM cpu report histograms:
  1. Choose DCM in the "Choose a Source" box
  2. Then, in the "Metric" box, choose cpu_report
  3. Then, if you want to see just the DCMs used in the current run, click on the Partition # of the run
  4. You can also choose the time included in the displays from the last hour to year!
  • Overview histograms are provided as well as cpu use for the DCMs in the selected Partition. For normal running, cpu use should be well under 100% for each DCM.

Nearline (bookmark)

On NOvA-DAQ-4
Web Apps:

  • FD, NDOS Cameras
  • Underground access FLUFFY and Permits display
  • IFBeam monitoring and VMon

On NOvA-DAQ-5
DCS Monitoring and Control of Detector Power Supplies

On leftmost monitor (of 4), browser should be displayed with a "Synoptic" tabs for both FarDet and NDOS and FarDet Index box (for selection of control pages)
On middle left monitor, NDOS display and FarDet HV should be seen (display pages, not control pages)
On middle right monitor, FarDet Low Voltage monitor should be displayed
On rightmost monitor, Environmental displays should be seen - Rack Monitor, Dry AIr and Chilled Water Monitor, Power Distribution, and FarDet Hall environmental monitor

*To get pages for controlling power supplies, click on the desired selection in the Index Box

A window looking very similar to the display page of the selection will appear, but will contain a connector icon at the bottom right corner
To access control, click on the connector icon

Commissioning Shift Checklist

Use FD Commissioning Report form in the ECL:

Select "New Entry"
In the pop-up window, select "FD Commissioning Report" in Form box
Select "Commissioning" in the Category box
Select "FD Commissioning" in the Tags box

Instructions for each form entry are included on the form

Snapshots of plots for report:

Use the snapshot icon in the toolbar to take a picture of plots
Save to folder ALL
This folder is accessible on all desktops - directory /tmp/ALL from insert picture on form

Commissioning Run Exec Summary