Project

General

Profile

Off-Site Procedures » History » Version 140

Ralitsa Sharankova, 10/02/2019 11:26 AM

1 66 Christoph Rudolf von rohr
h1. Off-Site Procedures
2 70 Christoph Rudolf von rohr
3 134 Sophie Berkman
Before familiarizing yourself with the procedures please read the [[Off-Site Shifting Requirements]].
4 66 Christoph Rudolf von rohr
5 66 Christoph Rudolf von rohr
{{toc}}
6 66 Christoph Rudolf von rohr
7 111 Glenn Horton-Smith
This page describes how to start up and perform your off-site shift.
8 111 Glenn Horton-Smith
For how to set up your off-site remote operations center, see [[Off-Site System Setup]].
9 111 Glenn Horton-Smith
10 66 Christoph Rudolf von rohr
h2. Off-Site Shifting Procedure
11 66 Christoph Rudolf von rohr
12 120 Sophie Berkman
In general an off-site shifter is obliged to follow the procedures described on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/. There are also some special procedures an off-site shifters will need to follow.
13 66 Christoph Rudolf von rohr
14 66 Christoph Rudolf von rohr
h3. Starting your Off-Site Shift
15 66 Christoph Rudolf von rohr
16 120 Sophie Berkman
First,arrange enough time to start up your off-site shift system. Under normal conditions it takes about 15 minutes to set everything up and you also want to have an overlap with previous shifter. (You are required to contact the previous shifter 10 minutes prior to the start of your shift block, remember).  You will also need time to follow the shift turnover procedures, including verifying that the call forwarding to your shift station is working.
17 66 Christoph Rudolf von rohr
18 66 Christoph Rudolf von rohr
* Follow the start-up or quick start-up guide below.
19 66 Christoph Rudolf von rohr
20 117 Ray Neely
* Get in touch with the previous shifter, usually via Slack, and exchange information on the detector status. Follow the [[Shift_turnover]] procedures.
21 66 Christoph Rudolf von rohr
22 66 Christoph Rudolf von rohr
* Exchange information with previous shifters.
23 66 Christoph Rudolf von rohr
24 92 Corey Adams
* Call MCR (use your landline). Inform them that you are off-site, and tell them what station you are at.
25 66 Christoph Rudolf von rohr
26 120 Sophie Berkman
%{background:yellow; color:black}If you encounter problems setting up your system, inform the Run Coordinator. In case of severe connection difficulties such that you cannot fulfill the requirements of shifting, you may also have to inform the backup shifter.%
27 66 Christoph Rudolf von rohr
28 66 Christoph Rudolf von rohr
h3. During your Off-Site Shift
29 66 Christoph Rudolf von rohr
30 120 Sophie Berkman
Although you may be thousands of miles away from the detector, it is very important that the communication between the shifters and experts works out well. Therefore please try to stick to these rules
31 1 Christoph Rudolf von rohr
32 120 Sophie Berkman
* Off-Site shifters must monitor the Slack chat carefully and should use it any time they have questions.
33 99 Pip Hamilton
34 120 Sophie Berkman
* If you need to step out, you can take quick breaks (few minutes) but if you'll be gone for longer arrange for shift coverage by someone who is also shift-trained. Ensure that anyone covering for you knows who to call in case of an emergency (experts and the run coordinator).
35 66 Christoph Rudolf von rohr
36 66 Christoph Rudolf von rohr
h2. Start-up Guide
37 66 Christoph Rudolf von rohr
38 120 Sophie Berkman
This guide lists all steps necessary for starting up your off-site remote shift system. All steps in this guide might change, so if steps in this guide do not work, please look up the information on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki.
39 66 Christoph Rudolf von rohr
40 66 Christoph Rudolf von rohr
h3. Establish Connection to FNAL
41 66 Christoph Rudolf von rohr
42 140 Ralitsa Sharankova
Get a Kerberos Ticket by using your FNAL computing account (e.g. kinit <username>@FNAL.GOV)
43 137 Sophie Berkman
44 140 Ralitsa Sharankova
Please remember to kinit **as yourself** at the start of each remote shift. Coasting on old credentials can cause you to lose connection when they unexpectedly expire.
45 1 Christoph Rudolf von rohr
46 121 Sophie Berkman
%{background:yellow; color:black}It is very important to regularly check that your VNC connections are up to date.%
47 100 Pip Hamilton
48 66 Christoph Rudolf von rohr
h3. Start-up of the Run Control VNC
49 66 Christoph Rudolf von rohr
50 66 Christoph Rudolf von rohr
Before you start up the run control VNC for the first time please check out [[RC - Guide]].
51 1 Christoph Rudolf von rohr
52 100 Pip Hamilton
Open a terminal/shell **with key forwarding enabled** (option -K)
53 66 Christoph Rudolf von rohr
<pre>
54 108 Pip Hamilton
ssh -K -L 5903:localhost:5903 uboonedaq@ubdaq-prod-ws01.fnal.gov
55 66 Christoph Rudolf von rohr
</pre>
56 66 Christoph Rudolf von rohr
and on ws01, do
57 1 Christoph Rudolf von rohr
<pre>
58 1 Christoph Rudolf von rohr
ssh -L 5903:localhost:5903 uboonedaq@ubdaq-prod-evb.fnal.gov
59 66 Christoph Rudolf von rohr
</pre>
60 102 Pip Hamilton
61 121 Sophie Berkman
%{background:yellow; color:black} *Do not kinit as yourself* on ws01 (or ws02, or evb). These machines have special principals; if you overwrite them you will lock the DAQ experts out of the system.%
62 1 Christoph Rudolf von rohr
63 121 Sophie Berkman
Now, connect your VNC client to localhost:5903. If you are using RealVNC, open a new terminal and start the VNC session locally on your machine with the command
64 66 Christoph Rudolf von rohr
<pre>
65 66 Christoph Rudolf von rohr
vncviewer localhost:5903
66 66 Christoph Rudolf von rohr
</pre>
67 110 Adrien Hourlier
*Use a separate terminal for this command, not in the ws01 or evb window!*
68 1 Christoph Rudolf von rohr
69 121 Sophie Berkman
Enter the standard DAQ password and wait for the VNC to appear.
70 66 Christoph Rudolf von rohr
71 66 Christoph Rudolf von rohr
If you are facing any problems read [[RC - Guide]] and [[RC - Troubleshooting]] before calling the expert.
72 66 Christoph Rudolf von rohr
73 1 Christoph Rudolf von rohr
h3. Start-up the Slow Monitoring/Control VNC
74 66 Christoph Rudolf von rohr
75 121 Sophie Berkman
If you are operating Slow Monitoring/Control for the first time make sure you read [[SLC - Guide]].
76 66 Christoph Rudolf von rohr
77 66 Christoph Rudolf von rohr
Open a terminal/shell
78 66 Christoph Rudolf von rohr
<pre>
79 127 Sophie Berkman
ssh -K -L 5902:localhost:5902 ubooneshift@ubdaq-prod-ws01.fnal.gov
80 66 Christoph Rudolf von rohr
</pre>
81 121 Sophie Berkman
now, connect your VNC client to localhost:5902. If you are using RealVNC, open a new terminal and start the VNC session locally on your machine with the command
82 66 Christoph Rudolf von rohr
<pre>
83 66 Christoph Rudolf von rohr
vncviewer localhost:5902
84 66 Christoph Rudolf von rohr
</pre>
85 66 Christoph Rudolf von rohr
Enter the standard uBooNE password+smc
86 66 Christoph Rudolf von rohr
87 66 Christoph Rudolf von rohr
If you face any issues while starting up or during operations check [[SLC - Guide]] and [[SLC - Troubleshooting]] before calling the expert.
88 66 Christoph Rudolf von rohr
89 1 Christoph Rudolf von rohr
h3. Start-up PUBS Monitoring GUI
90 66 Christoph Rudolf von rohr
91 66 Christoph Rudolf von rohr
For more information about PUBS read [[DM - Overview]] and [[DM - Shifters]]
92 66 Christoph Rudolf von rohr
93 66 Christoph Rudolf von rohr
Open a terminal/shell
94 66 Christoph Rudolf von rohr
<pre>
95 140 Ralitsa Sharankova
ssh -KX uboonedaq@ubdaq-prod-ws01.fnal.gov
96 66 Christoph Rudolf von rohr
</pre>
97 66 Christoph Rudolf von rohr
while on ws01 do
98 66 Christoph Rudolf von rohr
<pre>
99 81 Christoph Rudolf von rohr
ssh -X uboonedaq@ubdaq-prod-evb.fnal.gov
100 66 Christoph Rudolf von rohr
</pre>
101 66 Christoph Rudolf von rohr
on this machine execute the following lines
102 1 Christoph Rudolf von rohr
<pre>
103 1 Christoph Rudolf von rohr
cd pubs
104 1 Christoph Rudolf von rohr
source config/setup.sh
105 1 Christoph Rudolf von rohr
setup pyqtgraph
106 1 Christoph Rudolf von rohr
python $PUB_TOP_DIR/pub_mongui/mongui.py
107 1 Christoph Rudolf von rohr
</pre>
108 1 Christoph Rudolf von rohr
109 66 Christoph Rudolf von rohr
When problems occur follow [[DM - Troubleshooting]] before calling an expert.
110 1 Christoph Rudolf von rohr
111 121 Sophie Berkman
h3. Open a Remote Desktop Session for iFix
112 1 Christoph Rudolf von rohr
113 121 Sophie Berkman
To access iFix, you need to have previously activated your Fermi Domain account and arranged for it to be granted access to the ifix machine by contacting the Run Coordinators and providing your username.
114 1 Christoph Rudolf von rohr
-Once you have set up the usual VPN connection (above)- 
115 121 Sophie Berkman
116 82 Christoph Rudolf von rohr
Connect to ssh tunnel to iFix through ws01:
117 121 Sophie Berkman
<pre>
118 121 Sophie Berkman
ssh -C -N -K -L 13389:ppd-ifix2.fnal.gov:3389 ubooneshift@ubdaq-prod-ws01.fnal.gov
119 121 Sophie Berkman
</pre>
120 86 Corey Adams
121 140 Ralitsa Sharankova
Connect to iFix using a Remote Desktop Protocol client.  On Ubuntu, the default is Remmina. For Mac OSX one can use the Microsoft Remote Desktop app, instructions can be found here: https://technet.microsoft.com/en-us/library/dn473012(v=ws.11).aspx
122 121 Sophie Berkman
123 89 Corey Adams
The information you need for the connection using Remmina is: 
124 1 Christoph Rudolf von rohr
125 87 Corey Adams
* Protocol = RDP (Remote Desktop Protocol)
126 87 Corey Adams
* Server = -ppd-ifix2.fnal.gov- localhost:13389
127 121 Sophie Berkman
* Username = [leave this blank for now]
128 1 Christoph Rudolf von rohr
* Password = [leave this blank for now]
129 1 Christoph Rudolf von rohr
* Domain = [leave this blank]
130 1 Christoph Rudolf von rohr
131 1 Christoph Rudolf von rohr
132 140 Ralitsa Sharankova
Then, hit connect.  A Windows log in screen should load, and you will be prompted to enter your account information:
133 1 Christoph Rudolf von rohr
* Username = [your own personal Fermi Domain account]
134 1 Christoph Rudolf von rohr
* Password = [your own personal Fermi Domain password]
135 89 Corey Adams
136 140 Ralitsa Sharankova
To connect using Microsoft Remote Desktop use the following:
137 140 Ralitsa Sharankova
138 140 Ralitsa Sharankova
* Username = FERMI\< Fermi Domain username>
139 140 Ralitsa Sharankova
* Password = [leave blank]
140 140 Ralitsa Sharankova
141 140 Ralitsa Sharankova
Then, hit connect. A Windows log screen will appear:
142 140 Ralitsa Sharankova
* User: select your personal Fermi Domain username
143 140 Ralitsa Sharankova
* Password = [your personal Fermi Domain password]
144 89 Corey Adams
145 91 Corey Adams
You should see a windows computer load, and ifix will start automatically.  Be mindful to only observe the microboone alarm status, and do not make any alterations to any settings in the ifix.  When you first load the ifix, you may notice an message: "The alarm queue SUMMARY LIST has overflowed. At least one alarm may have been lost."  You can safely dismiss this message, the cryo experts have verified that none of the alarms are lost.  In general, if there are alarms right at the start up of the ifix session (red lights, warning messages, etc.) you can safely dismiss them.  
146 86 Corey Adams
147 66 Christoph Rudolf von rohr
h3. Open SSH Tunnel for Web Interfaces
148 66 Christoph Rudolf von rohr
149 84 Xiao Luo
In order to view the Online Monitoring Lizard, Ganglia and the PUBS status pages one needs to open the following SSH tunnel. This can be done in a separate Terminal by
150 66 Christoph Rudolf von rohr
<pre>
151 66 Christoph Rudolf von rohr
ssh -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov
152 66 Christoph Rudolf von rohr
</pre>
153 66 Christoph Rudolf von rohr
Now you are ready to start-up the web pages.
154 66 Christoph Rudolf von rohr
155 128 Sophie Berkman
Remember that for this to work you will need to have the browser settings using the automatic proxy: https://cdcvs.fnal.gov/redmine/attachments/download/24625/lartf.pac
156 128 Sophie Berkman
157 128 Sophie Berkman
158 66 Christoph Rudolf von rohr
h3. Open Web Pages
159 66 Christoph Rudolf von rohr
160 66 Christoph Rudolf von rohr
The following web pages need to be open during the whole shift.
161 66 Christoph Rudolf von rohr
162 66 Christoph Rudolf von rohr
In the Opera browser open
163 66 Christoph Rudolf von rohr
164 66 Christoph Rudolf von rohr
* "uBooNE Online Monitor Lizard":http://ubdaq-prod-near2.fnal.gov/lizard/
165 66 Christoph Rudolf von rohr
166 109 Adrien Hourlier
* "MicroBooNE Trigger Page":https://microboone-exp.fnal.gov/at_work/beam/performance/trigger.html
167 66 Christoph Rudolf von rohr
168 138 Glenn Horton-Smith
* "MicroBooNE Run Catalogue":http://ubdaq-prod-near2.fnal.gov/RunCat/web/
169 138 Glenn Horton-Smith
170 138 Glenn Horton-Smith
* "MicroBooNE Supernova Run Catalog":http://ubdaq-prod-near2.fnal.gov/RunCat/web/recent_sn.cgi
171 96 David Martinez
172 80 Christoph Rudolf von rohr
* "MicroBooNE Run Summary":http://ubdaq-prod-near2.fnal.gov/Pubs/RunSummary.html
173 79 Christoph Rudolf von rohr
174 80 Christoph Rudolf von rohr
* "MicroBooNE Run Statistics":http://ubdaq-prod-near2.fnal.gov/Pubs/RunStat.html
175 1 Christoph Rudolf von rohr
176 83 Christoph Rudolf von rohr
* "Network Weather Map":http://mrtg.fnal.gov/weathermap/wx-uboone.html
177 80 Christoph Rudolf von rohr
178 79 Christoph Rudolf von rohr
179 74 Christoph Rudolf von rohr
In any browser of your choice open
180 1 Christoph Rudolf von rohr
181 79 Christoph Rudolf von rohr
* "Ganglia Shifer View":http://ubdaq-prod-evb.fnal.gov:8080/gweb/?r=hour&cs=&ce=&tab=v&vn=shifter
182 1 Christoph Rudolf von rohr
183 79 Christoph Rudolf von rohr
* "MiniBooNE Monitor":http://www-boone.fnal.gov/monitor/
184 1 Christoph Rudolf von rohr
185 79 Christoph Rudolf von rohr
* "BNB Status Display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/bnbmon/Display
186 1 Christoph Rudolf von rohr
187 79 Christoph Rudolf von rohr
* "NuMi Status Display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/numimon/Display
188 1 Christoph Rudolf von rohr
189 79 Christoph Rudolf von rohr
* "Accelerator Division Notification Page":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top
190 66 Christoph Rudolf von rohr
191 79 Christoph Rudolf von rohr
* "Slow Monitoring Archiver Status":http://ubdaq-prod-smc.fnal.gov:4812/main
192 1 Christoph Rudolf von rohr
193 126 Sophie Berkman
* -"MicroBooNE Run Information Document":https://docs.google.com/spreadsheets/d/1aVGAZrCYizhmduY4pqx5k29pVjyPRO7ydBJj95i0XfA/edit#gid=0-
194 1 Christoph Rudolf von rohr
195 126 Sophie Berkman
* -"MicroBooNE Alarm Page":http://ubdaq-prod-smc.fnal.gov:4714/alarmpage.html-
196 66 Christoph Rudolf von rohr
197 79 Christoph Rudolf von rohr
* "MicroBooNE Operations Wiki":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki
198 66 Christoph Rudolf von rohr
199 79 Christoph Rudolf von rohr
* "MicroBooNE Run Plan":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Run_plan
200 66 Christoph Rudolf von rohr
201 119 Sophie Berkman
* "MicroBooNE Expert Call List":https://microboone-exp.fnal.gov/at_work/expert_call_list.html
202 79 Christoph Rudolf von rohr
203 117 Ray Neely
* "Shift Slack Channel":https://microboone.slack.com/messages/CHAD839HQ/
204 1 Christoph Rudolf von rohr
205 1 Christoph Rudolf von rohr
* "MicroBooNE Elog":http://dbweb0.fnal.gov/ECL/uboone/E/index
206 114 Glenn Horton-Smith
207 114 Glenn Horton-Smith
(Note the "PUBS Resource Monitoring":http://ubdaq-prod-near2.fnal.gov/Pubs/ page is "no longer needed":/redmine/projects/uboone-operations/wiki/OpsWikiUpdates#10th-April-2018 .)
208 66 Christoph Rudolf von rohr
209 66 Christoph Rudolf von rohr
If you face any problems with the online monitoring Lizard look into [[OM - Checklist]], [[OM - Guide]] and [[OM - Troubleshooting]].
210 66 Christoph Rudolf von rohr
211 66 Christoph Rudolf von rohr
When you have trouble with the BNB Beam monitor checkout [[Beam - Guide]].
212 66 Christoph Rudolf von rohr
213 66 Christoph Rudolf von rohr
h3. Communication with ROC West
214 66 Christoph Rudolf von rohr
215 94 Corey Adams
[We do not staff ROC west continuously anymore, so this section is no longer relevant.  It's kept here for reference in case we have period where we staff ROC west and a remote center again.]
216 94 Corey Adams
217 66 Christoph Rudolf von rohr
Communication with ROC West is an important part of your off-site shift, therefore:
218 66 Christoph Rudolf von rohr
219 78 Michael Mooney
* Connect to MicroBooNE's CERN Vidyo room using the instructions in [[Off-Site_System_Setup|Off-Site System Setup]].
220 66 Christoph Rudolf von rohr
221 117 Ray Neely
* Login to the MicroBooNE Slack and join the "shift" channel.
222 66 Christoph Rudolf von rohr
223 77 Michael Mooney
If the Vidyo connection fails, connect to Skype using your local shifting account and call ubooneshift.  If both fail, use your landline to communicate with ROC West. Also inform ROC West if you have trouble with your off-site system.
224 66 Christoph Rudolf von rohr
225 66 Christoph Rudolf von rohr
h3. Get ready for Shift
226 66 Christoph Rudolf von rohr
227 95 Corey Adams
If you successfully performed all of the above steps, you are *almost* ready for shift. Call MCR using your landline, give inform them about the off-site shift and leave them your phone number.
228 66 Christoph Rudolf von rohr
229 66 Christoph Rudolf von rohr
Now, you are ready for shift!
230 66 Christoph Rudolf von rohr
231 66 Christoph Rudolf von rohr
h2. Quick Start-up Guide
232 66 Christoph Rudolf von rohr
233 139 Sophie Berkman
-First start the VPN client using your FNAL services account credentials.-
234 112 Glenn Horton-Smith
235 112 Glenn Horton-Smith
Next open a terminal/shell
236 66 Christoph Rudolf von rohr
<pre>
237 66 Christoph Rudolf von rohr
ssh -L 5903:localhost:5903 -L 5902:localhost:5902 -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov
238 66 Christoph Rudolf von rohr
</pre>
239 66 Christoph Rudolf von rohr
and on ws01, do
240 66 Christoph Rudolf von rohr
<pre>
241 66 Christoph Rudolf von rohr
ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov
242 1 Christoph Rudolf von rohr
</pre>
243 1 Christoph Rudolf von rohr
244 112 Glenn Horton-Smith
Now connect yout VNC client to localhost:5903 and localhost:5902.
245 112 Glenn Horton-Smith
246 113 Glenn Horton-Smith
After that follow the steps to "start up PUBS":/redmine/projects/uboone-operations/wiki/Off-Site_Procedures#Start-up-PUBS-Monitoring-GUI above and open the all the needed pages with your browser(s).