Project

General

Profile

Off-Site Procedures » History » Version 78

« Previous - Version 78/150 (diff) - Next » - Current version
Michael Mooney, 01/27/2016 06:28 PM


Off-Site Procedures

Before familiarizing yourself with the procedures please read the Off-Site Shifting Requirements.

Off-Site Shifting Procedure

In general an off-site shifter is also obliged to follow the procedures described on the Wiki main page. Still there are some special procedures an off-site shifters need to follow in addition.

Starting your Off-Site Shift

First of all arrange enough time to start up your off-site shift system. Under normal conditions it takes about 15 minutes to set everything up and you also want to have an overlap with previous shifters.

  • Follow the start-up or quick start-up guide below.
  • Call ROC West using Vidyo, Skype or your landline.
  • Exchange information with previous shifters.
  • Even if the shifter in ROC West called MCR already you need to do so too (use your landline). Inform them that you are off-site and leave them your number.

If you run into problems while setting up your system, contact your co-shifter in ROC West and inform him about your problem. Maybe they even have the same issues. If you have severe connection problems inform the Run Coordinator.

During your Off-Site Shift

Although the shifters are separated by thousands of miles it is very important that the communication between the shifters works out well. Therefore try to stick to these rules

  • Off-Site shifters and ROC West shifters need to maintain at least one mean of communication.
  • Communicate all steps you are taking while facing alarms or unexpected detector behavior.
  • While troubleshooting, assign tasks, e.g. one shifter calls the expert the other writes an elog entry
  • Inform each other if you need to step out for a short time.

Start-up Guide

This guide lists all steps necessary for starting up your off-site remote shift system. All steps in this guide might be subject to changes If steps in this guide do not work, please look up the information on the Wiki main page.

Establish Connection to FNAL

  1. Start the VPN client. For this you need to use your FNAL services account credentials.
  2. Get a Kerberos Ticket by using your FNAL computing account (e.g. kinit <username>@FNAL.GOV)

It is very important to connect by VPN before any SSH tunnel is opened. If you do it later, you will end up having broken pipes and VNC freezes after some time.

Start-up of the Run Control VNC

Before you start up the run control VNC for the first time please check out RC - Guide.

Open a terminal/shell

ssh -L 5903:localhost:5903 uboonedaq@ubdaq-prod-ws01.fnal.gov

and on ws01, do
ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov

now, connect your VNC client to localhost:5903. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command

vncviewer localhost:5903

Enter standard DAQ password and wait for the VNC to appear.

If you are facing any problems read RC - Guide and RC - Troubleshooting before calling the expert.

Start-up the Slow Monitoring/Control VNC

Also here, if you are operating Slow Monitoring/Control for the first time make sure you read SLC - Guide.

Open a terminal/shell

ssh -K -L 5902:localhost:5902 ubooneshift@ubdaq-prod-ws01

now, connect your VNC client to localhost:5902. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command
vncviewer localhost:5902

Enter the standard uBooNE password+smc

If you face any issues while starting up or during operations check SLC - Guide and SLC - Troubleshooting before calling the expert.

Start-up PUBS Monitoring GUI

For more information about PUBS read DM - Overview and DM - Shifters

Open a terminal/shell

ssh -X uboonedaq@ubdaq-prod-ws01.fnal.gov

while on ws01 do
ssh -X uboonedaq@ubdaq-prod-smc.fnal.gov

on this machine execute the following lines
cd pubs
source config/setup.sh
python $PUB_TOP_DIR/pub_mongui/mongui.py

When problems occur follow DM - Troubleshooting before calling an expert.

Open SSH Tunnel for Web Interfaces

In order to view the Slow Monitoring Lizard, Ganglia and the PUBS status pages one needs to open the following SSH tunnel. This can be done in a separate Terminal by

ssh -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov

Now you are ready to start-up the web pages.

Open Web Pages

The following web pages need to be open during the whole shift.

In the Opera browser open

In any browser of your choice open

If you face any problems with the online monitoring Lizard look into OM - Checklist, OM - Guide and OM - Troubleshooting.

When you have trouble with the BNB Beam monitor checkout Beam - Guide.

Communication with ROC West

Communication with ROC West is an important part of your off-site shift, therefore:

  • Login to Google+ and join the MicroBooNE shift hangout.

If the Vidyo connection fails, connect to Skype using your local shifting account and call ubooneshift. If both fail, use your landline to communicate with ROC West. Also inform ROC West if you have trouble with your off-site system.

Get ready for Shift

If you successfully performed all of the above steps, you are almost ready for shift. Even if your co-shifter already called MCR you have to do so too. Call MCR using your landline, give inform them about the off-site shift and leave them your phone number.

Now, you are ready for shift!

Quick Start-up Guide

Open a terminal/shell

ssh -L 5903:localhost:5903 -L 5902:localhost:5902 -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov

and on ws01, do
ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov

Now connect yout VNC client to localhost:5903 and localhost:5902. After that follow the steps to start up PUBS above and open the all the needed pages with your browser(s).