Off-Site Procedures » History » Version 82
« Previous -
Version 82/150
(diff) -
Next » -
Current version
Christoph Rudolf von rohr, 03/08/2016 08:14 AM
Off-Site Procedures¶
Before familiarizing yourself with the procedures please read the Off-Site Shifting Requirements.
Off-Site Shifting Procedure¶
In general an off-site shifter is also obliged to follow the procedures described on the Wiki main page. Still there are some special procedures an off-site shifters need to follow in addition.
Starting your Off-Site Shift¶
First of all arrange enough time to start up your off-site shift system. Under normal conditions it takes about 15 minutes to set everything up and you also want to have an overlap with previous shifters.
- Follow the start-up or quick start-up guide below.
- Call ROC West using Vidyo, Skype or your landline.
- Exchange information with previous shifters.
- Even if the shifter in ROC West called MCR already you need to do so too (use your landline). Inform them that you are off-site and leave them your number.
If you run into problems while setting up your system, contact your co-shifter in ROC West and inform him about your problem. Maybe they even have the same issues. If you have severe connection problems inform the Run Coordinator.
During your Off-Site Shift¶
Although the shifters are separated by thousands of miles it is very important that the communication between the shifters works out well. Therefore try to stick to these rules
- Off-Site shifters and ROC West shifters need to maintain at least one mean of communication.
- Communicate all steps you are taking while facing alarms or unexpected detector behavior.
- While troubleshooting, assign tasks, e.g. one shifter calls the expert the other writes an elog entry
- Inform each other if you need to step out for a short time.
Start-up Guide¶
This guide lists all steps necessary for starting up your off-site remote shift system. All steps in this guide might be subject to changes If steps in this guide do not work, please look up the information on the Wiki main page.
Establish Connection to FNAL¶
- Start the VPN client. For this you need to use your FNAL services account credentials.
- Get a Kerberos Ticket by using your FNAL computing account (e.g. kinit <username>@FNAL.GOV)
It is very important to connect by VPN before any SSH tunnel is opened. If you do it later, you will end up having broken pipes and VNC freezes after some time.
Start-up of the Run Control VNC¶
Before you start up the run control VNC for the first time please check out RC - Guide.
Open a terminal/shell
ssh -L 5903:localhost:5903 uboonedaq@ubdaq-prod-ws01.fnal.gov
and on ws01, do
ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov
now, connect your VNC client to localhost:5903. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command
vncviewer localhost:5903
Enter standard DAQ password and wait for the VNC to appear.
If you are facing any problems read RC - Guide and RC - Troubleshooting before calling the expert.
Start-up the Slow Monitoring/Control VNC¶
Also here, if you are operating Slow Monitoring/Control for the first time make sure you read SLC - Guide.
Open a terminal/shell
ssh -K -L 5902:localhost:5902 ubooneshift@ubdaq-prod-ws01
now, connect your VNC client to localhost:5902. In case you are using RealVNC open a new terminal and start the VNC session locally on your machine with the command
vncviewer localhost:5902
Enter the standard uBooNE password+smc
If you face any issues while starting up or during operations check SLC - Guide and SLC - Troubleshooting before calling the expert.
Start-up PUBS Monitoring GUI¶
For more information about PUBS read DM - Overview and DM - Shifters
Open a terminal/shell
ssh -X uboonedaq@ubdaq-prod-ws01.fnal.gov
while on ws01 do
ssh -X uboonedaq@ubdaq-prod-evb.fnal.gov
on this machine execute the following lines
cd pubs source config/setup.sh setup pyqtgraph python $PUB_TOP_DIR/pub_mongui/mongui.py
When problems occur follow DM - Troubleshooting before calling an expert.
Open SSH Tunnel for Web Interfaces¶
In order to view the Slow Monitoring Lizard, Ganglia and the PUBS status pages one needs to open the following SSH tunnel. This can be done in a separate Terminal by
ssh -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov
Now you are ready to start-up the web pages.
Open Web Pages¶
The following web pages need to be open during the whole shift.
In the Opera browser open
In any browser of your choice open
If you face any problems with the online monitoring Lizard look into OM - Checklist, OM - Guide and OM - Troubleshooting.
When you have trouble with the BNB Beam monitor checkout Beam - Guide.
Communication with ROC West¶
Communication with ROC West is an important part of your off-site shift, therefore:
- Connect to MicroBooNE's CERN Vidyo room using the instructions in Off-Site System Setup.
- Login to Google+ and join the MicroBooNE shift hangout.
If the Vidyo connection fails, connect to Skype using your local shifting account and call ubooneshift. If both fail, use your landline to communicate with ROC West. Also inform ROC West if you have trouble with your off-site system.
Get ready for Shift¶
If you successfully performed all of the above steps, you are almost ready for shift. Even if your co-shifter already called MCR you have to do so too. Call MCR using your landline, give inform them about the off-site shift and leave them your phone number.
Now, you are ready for shift!
Quick Start-up Guide¶
Open a terminal/shell
ssh -L 5903:localhost:5903 -L 5902:localhost:5902 -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov
and on ws01, do
ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov
Now connect yout VNC client to localhost:5903 and localhost:5902. After that follow the steps to start up PUBS above and open the all the needed pages with your browser(s).