uBoone Operations

h1. Off-Site Procedures

Before familiarizing yourself with the procedures please read the [[Off-Site Shifting Requirements]].

{{toc}}

This page describes how to start up and perform your off-site shift.

For how to set up your off-site remote operations center, see [[Off-Site System Setup]].

h2. Off-Site Shifting Procedure

In general an off-site shifter is obliged to follow the procedures described on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/. There are also some special procedures an off-site shifters will need to follow.

h3. Starting your Off-Site Shift

First,arrange enough time to start up your off-site shift system. Under normal conditions it takes about 15 minutes to set everything up and you also want to have an overlap with previous shifter. (You are required to contact the previous shifter 10 minutes prior to the start of your shift block, remember).  You will also need time to follow the shift turnover procedures, including verifying that the call forwarding to your shift station is working.

* Follow the start-up or quick start-up guide below.

* Get in touch with the previous shifter, usually via Slack, and exchange information on the detector status. Follow the [[Shift_turnover]] procedures.

* Exchange information with previous shifters.

* Call MCR (use your landline). Inform them that you are off-site, and tell them what station you are at.

%{background:yellow; color:black}If you encounter problems setting up your system, inform the Run Coordinator. In case of severe connection difficulties such that you cannot fulfill the requirements of shifting, you may also have to inform the backup shifter.%

h3. During your Off-Site Shift

Although you may be thousands of miles away from the detector, it is very important that the communication between the shifters and experts works out well. Therefore please try to stick to these rules

* Off-Site shifters must monitor the Slack chat carefully and should use it any time they have questions.

* If you need to step out, you can take quick breaks (few minutes) but if you'll be gone for longer arrange for shift coverage by someone who is also shift-trained. Ensure that anyone covering for you knows who to call in case of an emergency (experts and the run coordinator).

h2. Start-up Guide

This guide lists all steps necessary for starting up your off-site remote shift system. All steps in this guide might change, so if steps in this guide do not work, please look up the information on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki.

h3. Establish Connection to FNAL

###. -# Start the VPN client. For this you need to use your FNAL services account credentials.-

# Get a Kerberos Ticket by using your FNAL computing account (e.g. kinit <username>@FNAL.GOV)

%{background:yellow; color:black}It is very important to regularly check that your VNC connections are up to date.%

-%{background:yellow; color:black}It is very important to connect by VPN *before* any SSH tunnel is opened. If you do it later, you will end up having broken pipes and VNC freezes after some time.%-

Please remember to kinit **as yourself** at the start of each remote shift. Coasting on old credentials can cause you to lose connection when they unexpectedly expire.

h3. Start-up of the Run Control VNC

Before you start up the run control VNC for the first time please check out [[RC - Guide]].

Open a terminal/shell **with key forwarding enabled** (option -K)

<pre>

ssh -K -L 5903:localhost:5903 uboonedaq@ubdaq-prod-ws01.fnal.gov

</pre>

and on ws01, do

<pre>

ssh -L 5903:localhost:5903 uboonedaq@ubdaq-prod-evb.fnal.gov

</pre>

%{background:yellow; color:black} *Do not kinit as yourself* on ws01 (or ws02, or evb). These machines have special principals; if you overwrite them you will lock the DAQ experts out of the system.%

Now, connect your VNC client to localhost:5903. If you are using RealVNC, open a new terminal and start the VNC session locally on your machine with the command

<pre>

vncviewer localhost:5903

</pre>

*Use a separate terminal for this command, not in the ws01 or evb window!*

Enter the standard DAQ password and wait for the VNC to appear.

If you are facing any problems read [[RC - Guide]] and [[RC - Troubleshooting]] before calling the expert.

h3. Start-up the Slow Monitoring/Control VNC

If you are operating Slow Monitoring/Control for the first time make sure you read [[SLC - Guide]].

Open a terminal/shell

<pre>

ssh -K -L 5902:localhost:5902 ubooneshift@ubdaq-prod-ws01.fnal.gov

</pre>

now, connect your VNC client to localhost:5902. If you are using RealVNC, open a new terminal and start the VNC session locally on your machine with the command

<pre>

vncviewer localhost:5902

</pre>

Enter the standard uBooNE password+smc

If you face any issues while starting up or during operations check [[SLC - Guide]] and [[SLC - Troubleshooting]] before calling the expert.

h3. Start-up PUBS Monitoring GUI

For more information about PUBS read [[DM - Overview]] and [[DM - Shifters]]

Open a terminal/shell

<pre>

ssh -X uboonedaq@ubdaq-prod-ws01.fnal.gov

</pre>

while on ws01 do

<pre>

ssh -X uboonedaq@ubdaq-prod-evb.fnal.gov

</pre>

on this machine execute the following lines

<pre>

cd pubs

source config/setup.sh

setup pyqtgraph

python $PUB_TOP_DIR/pub_mongui/mongui.py

</pre>

When problems occur follow [[DM - Troubleshooting]] before calling an expert.

h3. Open a Remote Desktop Session for iFix

To access iFix, you need to have previously activated your Fermi Domain account and arranged for it to be granted access to the ifix machine by contacting the Run Coordinators and providing your username.

-Once you have set up the usual VPN connection (above)- 

Connect to ssh tunnel to iFix through ws01:

<pre>

ssh -C -N -K -L 13389:ppd-ifix2.fnal.gov:3389 ubooneshift@ubdaq-prod-ws01.fnal.gov

</pre>

Connect to iFix using a Remote Desktop Protocol client.  On Ubuntu, the default is Remmina.  Some users have suggested the following page for guidelines for mac: https://technet.microsoft.com/en-us/library/dn473012(v=ws.11).aspx

The information you need for the connection using Remmina is: 

* Protocol = RDP (Remote Desktop Protocol)

* Server = -ppd-ifix2.fnal.gov- localhost:13389

* Username = [leave this blank for now]

* Password = [leave this blank for now]

* Domain = [leave this blank]

Then, hit connect.  A windows log in screen should load, and you will be prompted to enter your account information:

* Username = [your own personal Fermi Domain account]

* Password = [your own personal Fermi Domain password]

You should see a windows computer load, and ifix will start automatically.  Be mindful to only observe the microboone alarm status, and do not make any alterations to any settings in the ifix.  When you first load the ifix, you may notice an message: "The alarm queue SUMMARY LIST has overflowed. At least one alarm may have been lost."  You can safely dismiss this message, the cryo experts have verified that none of the alarms are lost.  In general, if there are alarms right at the start up of the ifix session (red lights, warning messages, etc.) you can safely dismiss them.  

h3. Open SSH Tunnel for Web Interfaces

In order to view the Online Monitoring Lizard, Ganglia and the PUBS status pages one needs to open the following SSH tunnel. This can be done in a separate Terminal by

<pre>

ssh -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov

</pre>

Now you are ready to start-up the web pages.

Remember that for this to work you will need to have the browser settings using the automatic proxy: https://cdcvs.fnal.gov/redmine/attachments/download/24625/lartf.pac

h3. Open Web Pages

The following web pages need to be open during the whole shift.

In the Opera browser open

* "uBooNE Online Monitor Lizard":http://ubdaq-prod-near2.fnal.gov/lizard/

* "MicroBooNE Trigger Page":https://microboone-exp.fnal.gov/at_work/beam/performance/trigger.html

* "MicroBooNE Run Catalogue":http://ubdaq-prod-near2.fnal.gov/RunCat/web/

* "MicroBooNE Supernova Run Catalog":http://ubdaq-prod-near2.fnal.gov/RunCat/web/recent_sn.cgi

* "MicroBooNE Run Summary":http://ubdaq-prod-near2.fnal.gov/Pubs/RunSummary.html

* "MicroBooNE Run Statistics":http://ubdaq-prod-near2.fnal.gov/Pubs/RunStat.html

* "Network Weather Map":http://mrtg.fnal.gov/weathermap/wx-uboone.html

In any browser of your choice open

* "Ganglia Shifer View":http://ubdaq-prod-evb.fnal.gov:8080/gweb/?r=hour&cs=&ce=&tab=v&vn=shifter

* "MiniBooNE Monitor":http://www-boone.fnal.gov/monitor/

* "BNB Status Display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/bnbmon/Display

* "NuMi Status Display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/numimon/Display

* "Accelerator Division Notification Page":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top

* "Slow Monitoring Archiver Status":http://ubdaq-prod-smc.fnal.gov:4812/main

* -"MicroBooNE Run Information Document":https://docs.google.com/spreadsheets/d/1aVGAZrCYizhmduY4pqx5k29pVjyPRO7ydBJj95i0XfA/edit#gid=0-

* -"MicroBooNE Alarm Page":http://ubdaq-prod-smc.fnal.gov:4714/alarmpage.html-

* "MicroBooNE Operations Wiki":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki

* "MicroBooNE Run Plan":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Run_plan

* "MicroBooNE Expert Call List":https://microboone-exp.fnal.gov/at_work/expert_call_list.html

* "Shift Slack Channel":https://microboone.slack.com/messages/CHAD839HQ/

* "MicroBooNE Elog":http://dbweb0.fnal.gov/ECL/uboone/E/index

(Note the "PUBS Resource Monitoring":http://ubdaq-prod-near2.fnal.gov/Pubs/ page is "no longer needed":/redmine/projects/uboone-operations/wiki/OpsWikiUpdates#10th-April-2018 .)

If you face any problems with the online monitoring Lizard look into [[OM - Checklist]], [[OM - Guide]] and [[OM - Troubleshooting]].

When you have trouble with the BNB Beam monitor checkout [[Beam - Guide]].

h3. Communication with ROC West

[We do not staff ROC west continuously anymore, so this section is no longer relevant.  It's kept here for reference in case we have period where we staff ROC west and a remote center again.]

Communication with ROC West is an important part of your off-site shift, therefore:

* Connect to MicroBooNE's CERN Vidyo room using the instructions in [[Off-Site_System_Setup|Off-Site System Setup]].

* Login to the MicroBooNE Slack and join the "shift" channel.

If the Vidyo connection fails, connect to Skype using your local shifting account and call ubooneshift.  If both fail, use your landline to communicate with ROC West. Also inform ROC West if you have trouble with your off-site system.

h3. Get ready for Shift

If you successfully performed all of the above steps, you are *almost* ready for shift. Call MCR using your landline, give inform them about the off-site shift and leave them your phone number.

Now, you are ready for shift!

h2. Quick Start-up Guide

-First start the VPN client using your FNAL services account credentials.-

Next open a terminal/shell

<pre>

ssh -L 5903:localhost:5903 -L 5902:localhost:5902 -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov

</pre>

and on ws01, do

<pre>

ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov

</pre>

Now connect yout VNC client to localhost:5903 and localhost:5902.

After that follow the steps to "start up PUBS":/redmine/projects/uboone-operations/wiki/Off-Site_Procedures#Start-up-PUBS-Monitoring-GUI above and open the all the needed pages with your browser(s).