Project

General

Profile

Off-Site Procedures » History » Version 74

Version 73 (Christoph Rudolf von rohr, 01/07/2016 09:15 AM) → Version 74/145 (Christoph Rudolf von rohr, 01/10/2016 12:28 AM)

h1. Off-Site Procedures

Before familiarizing yourself with the procedures please read the [[Off-Site Shifting Requirements]].

{{toc}}

h2. Off-Site Shifting Procedure

In general an off-site shifter is also obliged to follow the procedures described on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/. Still there are some special procedures an off-site shifters need to follow in addition.

h3. Starting your Off-Site Shift

First of all arrange enough time to start up your off-site shift system. Under normal conditions it takes about 15 minutes to set everything up and you also want to have an overlap with previous shifters.

* Follow the start-up or quick start-up guide below.

* Call ROC-West using Skype or your landline.

* Exchange information with previous shifters.

* Even if the shifter in ROC-West called MCR already you need to do so too (use your land line). Inform them that you are off-site and leave them your number.

%{background:yellow; color:black}If you run into problems while setting up your system, contact your co-shifter in ROC-West and inform him about your problem. Maybe they even have the same issues. If you have severe connection problems inform the run coordinator.%

h3. During your Off-Site Shift

Although the shifters are separated by thousands of miles it is very important that the communication between the shifters works out well. Therefore try to stick to these rules

* Off-Site shifters and ROC West shifters need to maintain at least one mean of communication.

* Communicate all steps you are taking while facing alarms or unexpected detector behavior.

* While troubleshooting, assign tasks, e.g. one shifter calls the expert the other writes an elog entry

* Inform each other if you need to step out for a short time.

h2. Start-up Guide

This guide lists all steps necessary for starting up your off-site remote shift system. All steps in this guide might be subject to changes If steps in this guide do not work, please look up the information on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki.

h3. Establish Connection to FNAL

# Start the VPN client. For this you need to use your FNAL services account credentials.
# Get a Kerberos Ticket by using your FNAL computing account (e.g. kinit <username>@FNAL.GOV)

%{background:yellow; color:black}It is very important to connect by VPN *before* any SSH tunnel is opened. If you do it later, you will end up having broken pipes and VNC freezes after some time.%

h3. Start-up of the Run Control VNC

Before you start up the run control VNC for the first time please check out [[RC - Guide]].

Open a terminal/shell
<pre>
ssh -L 5903:localhost:5903 uboonedaq@ubdaq-prod-ws01.fnal.gov
</pre>
and on ws01, do
<pre>
ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov
</pre>

now, connect your VNC client to localhost:5903. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command
<pre>
vncviewer localhost:5903
</pre>

Enter standard DAQ password and wait for the VNC to appear.

If you are facing any problems read [[RC - Guide]] and [[RC - Troubleshooting]] before calling the expert.

h3. Start-up the Slow Monitoring/Control VNC

Also here, if you are operating Slow Monitoring/Control for the first time make sure you read [[SLC - Guide]].

Open a terminal/shell
<pre>
ssh -K -L 5902:localhost:5902 ubooneshift@ubdaq-prod-ws01
</pre>
now, connect your VNC client to localhost:5902. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command
<pre>
vncviewer localhost:5902
</pre>
Enter the standard uBooNE password+smc

If you face any issues while starting up or during operations check [[SLC - Guide]] and [[SLC - Troubleshooting]] before calling the expert.

h3. Start-up PUBS Monitoring GUI

For more information about PUBS read [[DM - Overview]] and [[DM - Shifters]]

Open a terminal/shell
<pre>
ssh -X uboonedaq@ubdaq-prod-ws01.fnal.gov
</pre>
while on ws01 do
<pre>
ssh -X uboonedaq@ubdaq-prod-smc.fnal.gov
</pre>
on this machine execute the following lines
<pre>
cd pubs
source config/setup.sh
python $PUB_TOP_DIR/pub_mongui/mongui.py
</pre>

When problems occur follow [[DM - Troubleshooting]] before calling an expert.

h3. Open SSH Tunnel for Web Interfaces

In order to view the Slow Monitoring Lizard, Ganglia and the PUBS status pages one needs to open the following SSH tunnel. This can be done in a separate Terminal by
<pre>
ssh -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov
</pre>
Now you are ready to start-up the web pages.

h3. Open Web Pages

The following web pages need to be open during the whole shift.

In the Opera browser open

* "uBooNE Online Monitor Lizard":http://ubdaq-prod-near2.fnal.gov/lizard/

* "PUBS Resource Monitoring":http://ubdaq-prod-near2.fnal.gov/Pubs/

In any browser of your choice open

* "Ganglia shifer view":http://ubdaq-prod-evb.fnal.gov:8080/gweb/?r=hour&cs=&ce=&tab=v&vn=shifter

* "MiniBoone monitor":http://www-boone.fnal.gov/monitor/

* "BNB status display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/bnbmon/Display display":http://dbweb4.fnal.gov:8080/ifbeam/bmon/bnbmon/Display

* "NuMi status display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/numimon/Display display":http://dbweb4.fnal.gov:8080/ifbeam/bmon/numimon/Display

* "Accelerator division notification page":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top

* "Slow monitoring archiver status":http://ubdaq-prod-smc.fnal.gov:4812/main

* "MicroBooNE run information document":https://docs.google.com/spreadsheets/d/1aVGAZrCYizhmduY4pqx5k29pVjyPRO7ydBJj95i0XfA/edit#gid=0

* "MicroBooNE operations wiki":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki

* "MicroBooNE run plan":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Run_plan

* "MicroBooNE expert call list":http://www-microboone.fnal.gov/at_work/expert_call_list.html

* "Google+ Hangout":https://plus.google.com/

* "MicroBooNE elog":http://dbweb0.fnal.gov/ECL/uboone/E/index

If you face any problems with the online monitoring Lizard look into [[OM - Checklist]], [[OM - Guide]] and [[OM - Troubleshooting]].

When you have trouble with the BNB Beam monitor checkout [[Beam - Guide]].

h3. Communication with ROC West

Communication with ROC West is an important part of your off-site shift, therefore:

* Connect to skype using your local shifting account and call ubooneshift.

* Login to google+ and join the MicroBooNE shift hangout.

If the skype connection fails, use your landline to communicate with ROC West. Also inform ROC West if you have trouble with your off-site system.

h3. Get ready for Shift

If you successfully performed all of the above steps, you are *almost* ready for shift. Even if your co-shifter already called MCR you have to do so too. Call MCR using your landline, give inform them about the off-site shift and leave them your phone number.

Now, you are ready for shift!

h2. Quick Start-up Guide

Open a terminal/shell
<pre>
ssh -L 5903:localhost:5903 -L 5902:localhost:5902 -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov
</pre>
and on ws01, do
<pre>
ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov
</pre>

Now connect yout VNC client to localhost:5903 and localhost:5902. After that follow the steps to start up PUBS above and open the all the needed pages with your browser(s).