Off-Site Procedures » History » Version 79

Version 78 (Michael Mooney, 01/27/2016 06:28 PM) → Version 79/150 (Christoph Rudolf von rohr, 03/08/2016 08:10 AM)

h1. Off-Site Procedures

Before familiarizing yourself with the procedures please read the [[Off-Site Shifting Requirements]].


h2. Off-Site Shifting Procedure

In general an off-site shifter is also obliged to follow the procedures described on the "Wiki main page": Still there are some special procedures an off-site shifters need to follow in addition.

h3. Starting your Off-Site Shift

First of all arrange enough time to start up your off-site shift system. Under normal conditions it takes about 15 minutes to set everything up and you also want to have an overlap with previous shifters.

* Follow the start-up or quick start-up guide below.

* Call ROC West using Vidyo, Skype or your landline.

* Exchange information with previous shifters.

* Even if the shifter in ROC West called MCR already you need to do so too (use your landline). Inform them that you are off-site and leave them your number.

%{background:yellow; color:black}If you run into problems while setting up your system, contact your co-shifter in ROC West and inform him about your problem. Maybe they even have the same issues. If you have severe connection problems inform the Run Coordinator.%

h3. During your Off-Site Shift

Although the shifters are separated by thousands of miles it is very important that the communication between the shifters works out well. Therefore try to stick to these rules

* Off-Site shifters and ROC West shifters need to maintain at least one mean of communication.

* Communicate all steps you are taking while facing alarms or unexpected detector behavior.

* While troubleshooting, assign tasks, e.g. one shifter calls the expert the other writes an elog entry

* Inform each other if you need to step out for a short time.

h2. Start-up Guide

This guide lists all steps necessary for starting up your off-site remote shift system. All steps in this guide might be subject to changes If steps in this guide do not work, please look up the information on the "Wiki main page":

h3. Establish Connection to FNAL

# Start the VPN client. For this you need to use your FNAL services account credentials.
# Get a Kerberos Ticket by using your FNAL computing account (e.g. kinit <username>@FNAL.GOV)

%{background:yellow; color:black}It is very important to connect by VPN *before* any SSH tunnel is opened. If you do it later, you will end up having broken pipes and VNC freezes after some time.%

h3. Start-up of the Run Control VNC

Before you start up the run control VNC for the first time please check out [[RC - Guide]].

Open a terminal/shell
ssh -L 5903:localhost:5903
and on ws01, do
ssh -L 5903:localhost:5903

now, connect your VNC client to localhost:5903. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command
vncviewer localhost:5903

Enter standard DAQ password and wait for the VNC to appear.

If you are facing any problems read [[RC - Guide]] and [[RC - Troubleshooting]] before calling the expert.

h3. Start-up the Slow Monitoring/Control VNC

Also here, if you are operating Slow Monitoring/Control for the first time make sure you read [[SLC - Guide]].

Open a terminal/shell
ssh -K -L 5902:localhost:5902 ubooneshift@ubdaq-prod-ws01
now, connect your VNC client to localhost:5902. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command
vncviewer localhost:5902
Enter the standard uBooNE password+smc

If you face any issues while starting up or during operations check [[SLC - Guide]] and [[SLC - Troubleshooting]] before calling the expert.

h3. Start-up PUBS Monitoring GUI

For more information about PUBS read [[DM - Overview]] and [[DM - Shifters]]

Open a terminal/shell
ssh -X
while on ws01 do
ssh -X
on this machine execute the following lines
cd pubs
source config/
python $PUB_TOP_DIR/pub_mongui/

When problems occur follow [[DM - Troubleshooting]] before calling an expert.

h3. Open SSH Tunnel for Web Interfaces

In order to view the Slow Monitoring Lizard, Ganglia and the PUBS status pages one needs to open the following SSH tunnel. This can be done in a separate Terminal by
ssh -D 8881
Now you are ready to start-up the web pages.

h3. Open Web Pages

The following web pages need to be open during the whole shift.

In the Opera browser open

* "uBooNE Online Monitor Lizard":

* "PUBS Resource Monitoring":

* "MicroBooNE Trigger Page":

* "MicroBooNE Run Summary"

* "MicroBooNE Run Statistics"

* "Weather Map":

In any browser of your choice open

* "Ganglia Shifer View": shifer view":

* "MiniBooNE Monitor": "MiniBoone monitor":

* "BNB Status Display": status display":

* "NuMi Status Display": status display":

* "Accelerator Division Notification Page": division notification page":

* "Slow Monitoring Archiver Status": monitoring archiver status":

* "MicroBooNE Run Information Document": run information document":

* "MicroBooNE Alarm Page": operations wiki":

* "MicroBooNE Operations Wiki": run plan":

* "MicroBooNE Run Plan": expert call list":

* "MicroBooNE Expert Call List":

"Google+ Hangout":

* "MicroBooNE Elog": elog":

If you face any problems with the online monitoring Lizard look into [[OM - Checklist]], [[OM - Guide]] and [[OM - Troubleshooting]].

When you have trouble with the BNB Beam monitor checkout [[Beam - Guide]].

h3. Communication with ROC West

Communication with ROC West is an important part of your off-site shift, therefore:

* Connect to MicroBooNE's CERN Vidyo room using the instructions in [[Off-Site_System_Setup|Off-Site System Setup]].

* Login to Google+ and join the MicroBooNE shift hangout.

If the Vidyo connection fails, connect to Skype using your local shifting account and call ubooneshift. If both fail, use your landline to communicate with ROC West. Also inform ROC West if you have trouble with your off-site system.

h3. Get ready for Shift

If you successfully performed all of the above steps, you are *almost* ready for shift. Even if your co-shifter already called MCR you have to do so too. Call MCR using your landline, give inform them about the off-site shift and leave them your phone number.

Now, you are ready for shift!

h2. Quick Start-up Guide

Open a terminal/shell
ssh -L 5903:localhost:5903 -L 5902:localhost:5902 -D 8881
and on ws01, do
ssh -L 5903:localhost:5903

Now connect yout VNC client to localhost:5903 and localhost:5902. After that follow the steps to start up PUBS above and open the all the needed pages with your browser(s).