FD Commissioning » History » Version 56

Version 55 (Luke Corwin, 06/12/2013 12:50 PM) → Version 56/107 (Luke Corwin, 06/13/2013 03:44 PM)

h1. FD Commissioning

h2. Specific instructions for shifters involved in commissioning of the Far Detector

> Included are specific instructions for performing shift duties related to Far Detector commissioning.
> Updates are included regularly after evaluation of the shifter commissioning experience.

General schedule for commissioning shifts (daytime hours):
> 0800-1000 Evaluate overnight commissioning run using commissioning shift checklist
> 1000-1500 Monitor ongoing NDOS, FD runs, perform DSO scans, short test runs as requested by FD outfitters
> 1500-1700 Start overnight FD commissioning run with new detector elements

h2. General procedures to be aware of on shift
> Calls from Ash River outfitters
> > Acknowledge the message of the call and make an entry in the ECL

> > Example : WH control Room will get a call when a group of APDs has been installed and has been turned over for commissioning operations
> > > Acknowledge the message and enter into the logbook the DCM designation(s)
> > > Call Expert to have evaluation of APD environment done (Leon Mualem or Mat Muether)
> > > Once approved follow [[instructions]] for ramping of HV and recording of DSO scans
> > > On completion, enter into logbook status of the channels, i.e., ready for HV operation

At times one may want to put a link in one logbook (ECL) to another logbook.
> First, make the entry in the first ECL
> Click the bold blue header for that entry in the first ECL; this will take you to a web page for that entry.
> Right click the hyperlink beside "URL:" on this page.
> Select Copy Link Location
> Go two the second ECL
> Click New Entry
> In the new entry, click the box beside "Textile formatted:" and under "Address"
> Scroll down to the "Insert link:" box.
> Paste the URL into the box
> In the box under "Display," type whatever words you want to link to the entry in the first ECL
> Click "insert" (to the right of the "Display" box). This will insert code into the "Entry text:" box
> Type the remainder of your new entry around the link.
> Click "Post" to post the entry into the second ECL.

h2. Control Room Configuration

First, you will notice 5 groups of monitors in CR:
> NOVA-DAQ-1: Start and Stop Run Control, Pedestal Runner
> NOVA-DAQ-2: Memory Viewer, Event Display, APD Temperature Monitor, OnMon
> NOVA-DAQ-3: Ganglia, Nearline
> NOVA-DAQ-4: FD, NDOS cameras
> NOVA-DAQ-5: DCS Monitoring and Control
On any group, web applications can be obtained by starting FireFox from the toolbar
Click on Home to get standard web pages for shift Operations
Bookmarks include :
> NOvA ECL (Electronic Collaboration Logbook), Ganglia Monitoring Program, Ash River ECL, Nearline Monitoring, FD Commissioning Shift Instructions

h3. Switching Between NDOS and FD Monitoring in VNC :

Each of the machines usually has two virtual network connections (VNCs) running on it, one for the NDOS and one for the FD. What you usually see when first looking at the screens is actually one of these VNC connection. The FD VNC has a maroon background, and the NDOS has a black background.

Hit F8 for gui - click to exit the Full Screen mode.
NDOS is usually in session 1 and FD is usually in session 3. Choose the session you want to monitor, hit F8 and switch back to the full screen mode.
If no session exists for the detector you want, click on the appropriate icon (usually at top left) to start a new session for that detector

h3. In case of VNC trouble

If you cannot get a VNC started using the appropriate icon, the VNC server may need to be restarted. Do this on novadaq-far-master-02 by issuing the command: <pre></pre>

For more info on the VNCs, see [[]], [[]], and NOvA-doc-8406.

h2. Expert Notification Procedure

Options : phone, email, logbook

h3. For Shifters :
> When a problem is encountered :
* Look up in Redmine pages
* solve problem with instructions
> *Enter into Logbook*
> > or
* contact expert :
# *Phone*
# if no answer on phone :
*Email Expert*
*Phone Run Coordinator*
# *Enter into Logbook*

h3. For Experts performing work :
> When starting work :
# *Phone Control Room (630-840-4008)*
# *Enter into Logbook*
> When work is complete or stopped :
# *Phone Control Room*
# *Enter into Logbook*

h2. On NOVA-DAQ-1

Desktop Icons :
> StartRunControl
> StopRunControl
> Run Pedestal Commissioning (specifically for commissioning work)
> Run Pedestal (reserved for normal operations after commissioning)
Displays include: Run Control, Message Facility, Trigger scalers, DAQ Resource Manager, DAQ AppMan, TDU for active runs

h3. To start a DSO Scan :

Click on Run Pedestal Commissioning icon on Desktop (or Run Pedestal for normal data-taking after commissioning)
> 2 windows will appear - a terminal window with scrolling messages and the Run Pedestal gui
> click "Go" to start scan

please check here for more information (old instructions for NDOS) :

To view results:
* Click on Check DSO Result - vs Database icon on Desktop
* Results will pop up in OnMon Viewer
* Make a screenshot snapshot of the Pixel Enable and Threshold results and post in the logbook
* In addition the raw results, we want the difference plots between the new results and those in the database. To do this, click on the Comparison tab. Select the Comparison and Difference radio buttons. The click, "Apply" Make the same screenshots as above for the new histograms.

If one does not wish to deal with taking screen shots, there is an alternate method of getting the plots into the logbook:
> In the OnMon view, bring up the plot you wish to save.
> Right click in the white area of the plot; this will bring up the TCanvas menu.
> Select SaveAs
> Input name into the "filename" field of the the dialog box that appears, preferably a name that is descriptive and unique and with a .png extension (for example DSOThresholds_FD__2013-05-23_09_28_14.png).
> Click OK. This will save the file into the directory ~novadaq/plotTestOffline/ You can either load them into the logbook from this directory directly or move them to /tmp/ALL as follows:
> Open a terminal on novadaq-far-master-02 (note this is *not* novadaq-far-master) The easiest way to do this is just open a fresh terminal.
> cp ~novadaq/plotTestOffline/filename.png /tmp/ALL/
> Then, you can post in the logbook as above.

To load Pixel Masks and Thresholds:
> FD - log onto novadaq-far-master as novadaq (ssh novadaq@novadaq-far-master)
> NDOS - log onto novadaq-ctrl-master as novadaq (ssh novadaq@novadaq-ctrl-master)
> then setup_online
> then type <date string>
> Example:
> [novadaq@novadaq-far-master ~]$ 2013-03-22_14_54_38

> For the FD, you can also use the DAQConfigEditor (launched from ICON on control machine 1)
> File-> Open
> Select FarDetGlobalConfigP1 and click OK (this may take a while to load)
> Click Action -> Load Thresholds and Mask
> The directory data1-a/PedestalData/PedScans should already be higlighted
> Expand the directory PedScans, and you will see a list of directories in the date format YYYY-MM-DD_HH_MM_SS
> Select the most recent directory and click OK
> For now, load the values with exception overrides. This may also take a while, but then you are done

h3. In the case of running DSO scans for APD Checkout or for some other unusual subset of the detector:

Edit the file

<pre> /nova/config/FarDet/pedestals/PedestalConfiguration_Standard_FarDet_Commissioning.xml </pre>

Uncomment or insert all DCMs to be included in the commissioning DSO scan; follow the pattern already in the file.

Then, double click the icon "Run Pedestal Commissioning," and proceed with the steps above, but do *not* load the pixel masks or thresholds from a commissioning DSO scan.

h2. Adjust DCM Voltages Using Voltage Divider Settings:

The voltage on the DCMs are set by a voltage divider that reduces the voltage from the power supply. Normally, the 425V from the supply is reduced to 349V (when warm) and 337V (when APDs are cooled). The voltage divider uses arbitrary units that are linearly related to the actual voltages. To adjust the voltage for a given DCM or set of DCMs, we adjust the voltage divider settings.

* Start by opening the DCS Config Editor from the icon on NOvA-DAQ-1
* Select the tab apd_high_voltages
* Expand the "warm" drop-down menu
* If the DCM you wish to adjust is not listed, right-click on the list and select "Add DCM." Add the DCM you want and click "OK"
* For the DCM you wish to adjust, double-click the number in the "voltage" column and enter the new value based (93 = 300 V, 1066 = 325 V). The nominal value is 2000, which corresponds to 349V.
* File -> Save
* Click yes in the pop-up window
* Action-> Generate Recipe Scripts
* Select ok (press return) in the confirmation window.
* To load these new settings reboot the dcms.
* Verify the settings were properly in the CSS GUI on the single APD view.

h3. Voltage Divider Calculations

h2. Start a Run:

Make sure no run is going in the partition you plan to use before you start a new run.
Double-click on the icon StartRunControl on the top left of NOVA-DAQ-1 screen
> automatically gets a kerberos ticket
> a dialog box appears - select Partition # and click OK
> Run Control gui appears on the Desktop with Discover Resources box green

Ready for starting a run :
1. Click on Discover Resources
> text changes to "Rediscover Resources" and Select Resources box also turns green
2. Click on Select Resources
> Resource Manager window appears on Desktop
> Check to make sure the resources you want for this run are selected
> if changes have been made, red notation will be present in some of the expanded selections
> > if this is the case, a green window appears at the bottom - Acknowledge All Deviations - click this if correct
> click OK
> Reserver Resources box on RC gui turns green
3. Click on Reserve Resources
> DAQApplicationManager appears on Desktop, wait until Managers, DCMApplications, and Buffer Nodes boxes turn green
> Trigger Scalars, Message Facility, and TDU Control Interface windows appear on Desktop
> > click OK in the TDUControl dialog box indicating that "FIRMWARE NOT SUPPORTED"
> Release Resources and Select Configuration boxes turn green
4. click on Select Configuration
> In the dialog box that appears, choose the latest configuration file by highlighting it, then click OK
> > If for specific reason you want to roll back to a certain configuration, click the "more" option and choose the one you want.
> Note: when you do this, make a log entry noting which configuration you use in the run, e.g., 2013-05-03_20_03_35 - the time stamp
> some messages appear in the Message facility (should be green (informational messages)
> Prepare Configuration box on RC gui turns green
5. Click on Prepare Configuration
> more green messages on Message Facility
> Load Connections box on RC gui turns green
6. Click on Load Connections
> more green messages on Message Facility
> Make Connections box on RC gui turns green
7. Click Make Connections
> more green messages on Message Facility
> Load Hardware Config. box on RC gui turns green
8. CLick Load Hardware Config.
> more green messages
> Configure Hardware box on RC gui turns green
9. Click Configure Hardware
> lots of green messages on Message Facility
> > if errors, yellow and/or red messages appear on Message Facility
> > Can see problem on DAQAppMan window
> > click Abort in ConfiguringHW dialog box
> Load Run Config. box on RC gui turns green
10. Click Load Run Config
> more green messages
> Configure Run box on RC gui turns green
11. Click Configure Run
> more green messages
> Begin Run box on RC gui turns green
12. Before clicking Begin Run,

If starting an NDOS run:
> in a terminal window of novadaq-ctrl-master, do (example with 8 dcms):
<pre> -l root "dcm-3-01-{01-02},dcm-3-{02-03}-{01-03}" 'pid=` ps aux | awk ' " ' " ' /dcm[I]OC/{print$2} ' " ' " '`; kill -STOP $pid'</pre>

Click Begin Run
> Select type of run in dialog box and add a comment if you like, then click OK
> messages appear, Trigger Scalars start counting

> then in the terminal window:
<pre> -l root "dcm-3-01-{01-02},dcm-3-{02-03}-{01-03}" 'pid=` ps aux | awk ' " ' " ' /dcm[I]OC/{print$2} ' " ' " '`; kill -CONT $pid'

If starting a FD run:

> It is helpful to define an environment variable with your DCM list, for example, in a terminal window of window of novadaq-far-master:

<pre> dcmlist="dcm-2-02-0{1-6},dcm-2-03-{01-12}" </pre>

> Then, do:
<pre> -l root "$dcmlist" 'pid=`ps aux|awk '"'"'/dcm[I]OC/{print$2}'"'"'`; kill -STOP $pid'

Click Begin Run
> Select type of run in dialog box and add a comment if you like, then click OK
> messages appear, Trigger Scalars start counting

> then in the terminal window:
<pre> -l root "$dcmlist" 'pid=`ps aux|awk '"'"'/dcm[I]OC/{print$2}'"'"'`; kill -CONT $pid'

13. Verify that RC Start Run entry has been automatically made in ECL, if not, make an entry indicating that a new commissioning run has started.
14. On NOVA-DAQ-2, get a kerberos ticket, then start
> Memory Viewer using Desktop icon (use shared memory P000 for FarDet, P001 for NDOS)
> > click Connect to Memory
> Event Display using Desktop icon
> > in dialog box, select Partition # and click OK
> > start automatic scrolling by clicking on ">" at top of EVD
> Online Monitoring (OnMon)
> > click OnMon Producer desktop icon
> > > select Partition # in dialog box and click OK
> > > verify that events are scrolling in the producer window
> > click OnMon Viewer to start histogram viewer
> > > select Partition number in dialog box and click OK
> > > OnMon window appears
> > select histograms to view

h2. End a Run:

On RC gui, Pause Run and End Run boxes should be green
1. Click on End Run box on RC gui
> Comment on why run ended - click OK
> Release Resources, Select Configuration, Load Connections, and Begin Run boxes should turn green
2. Click Release Resources on RC gui
> Trigger Scalars, TDU COntrol, Message Facility, Resource Manager windows disappear
> DAQ Application Manager shows pink boxes for DCMs and Buffer Nodes, green/pink on Managers box
> Rediscover Resources, Select Resources, and Reserve Resources on RC gui are all green
> If this doesn't work, you will have to click "Abort" in a popup box
> > then click "Stop System" in DAQ App Manager Gui
3. Click Stop Run Control icon on desktop
> select Partition # in dialog box and click OK
> RC gui, DAQ APPManager windows disappear
4. Ready for new run

h2. On NOvA-DAQ-2

Desktop icons:

> *Event Display*
> > 2 icons on desktop - "ART Event Display" and "ART Event Display - STOP"

> *Online Monitoring (OnMon Producer, Viewer)*
> > 2 icons on desktop for Producer - "OnMon Producer" and "OnMon - STOP"
> > 2 icons on desktop for Viewer - "OnMon Viewer" and "OnMon Viewer - STOP"
> *"Online Monitoring Shifter Wiki":*

> *Memory Viewer*
> > start with "Memory Viewer" icon on desktop
> > stop by closing memory viewer window

> *APD Temperature Monitor*
> > 2 icons on desktop - "APD Temp Monitor" and "APD Temp Monitor - STOP"

> *DSO Results, DSO vs Reference*
> > desktop icon "Check DSO Results" displays latest DSO result through OnMon Viewer
> > desktop icon "Check DSO Result vs Database" displays latest DSO result and compares it to previous result also through OnMon Viewer

h2. On NOvA-DAQ-3
Web Apps:

> *Ganglia* (bookmarks for FD and NDOS)
> Very good monitor of the state of DCMs. To display DCM cpu report histograms:
> # Choose DCM in the "Choose a Source" box
> # Then, in the "Metric" box, choose cpu_report
> # Then, if you want to see just the DCMs used in the current run, click on the Partition # of the run
> # You can also choose the time included in the displays from the last hour to year!

> Overview histograms are provided as well as cpu use for the DCMs in the selected Partition. For normal running, cpu use should be well under 100% for each DCM.

> *Nearline* (bookmark)

h2. On NOvA-DAQ-4
Web Apps:
> *FD, NDOS Cameras*

h2. On NOvA-DAQ-5
*DCS Monitoring and Control of Detector Power Supplies*

h2. Commissioning Shift Checklist

Use FD Commissioning Report form in the ECL:
> Select "New Entry"
> In the pop-up window, select "FD Commissioning Report" in Form box
> Select "Commissioning" in the Category box
> Select "FD Commissioning" in the Tags box
Instructions for each form entry are included on the form

h3. Snapshots of plots for report:
> Use the snapshot icon in the toolbar to take a picture of plots
> Save to folder ALL
> This folder is accessible on all desktops - directory /tmp/ALL from insert picture on form

h2. Commissioning Run Exec Summary