Off-Site Procedures » History » Version 142
Ralitsa Sharankova, 10/03/2019 03:07 PM
1 | 66 | Christoph Rudolf von rohr | h1. Off-Site Procedures |
---|---|---|---|
2 | 70 | Christoph Rudolf von rohr | |
3 | 134 | Sophie Berkman | Before familiarizing yourself with the procedures please read the [[Off-Site Shifting Requirements]]. |
4 | 66 | Christoph Rudolf von rohr | |
5 | 66 | Christoph Rudolf von rohr | {{toc}} |
6 | 66 | Christoph Rudolf von rohr | |
7 | 111 | Glenn Horton-Smith | This page describes how to start up and perform your off-site shift. |
8 | 111 | Glenn Horton-Smith | For how to set up your off-site remote operations center, see [[Off-Site System Setup]]. |
9 | 111 | Glenn Horton-Smith | |
10 | 66 | Christoph Rudolf von rohr | h2. Off-Site Shifting Procedure |
11 | 66 | Christoph Rudolf von rohr | |
12 | 120 | Sophie Berkman | In general an off-site shifter is obliged to follow the procedures described on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/. There are also some special procedures an off-site shifters will need to follow. |
13 | 66 | Christoph Rudolf von rohr | |
14 | 66 | Christoph Rudolf von rohr | h3. Starting your Off-Site Shift |
15 | 66 | Christoph Rudolf von rohr | |
16 | 120 | Sophie Berkman | First,arrange enough time to start up your off-site shift system. Under normal conditions it takes about 15 minutes to set everything up and you also want to have an overlap with previous shifter. (You are required to contact the previous shifter 10 minutes prior to the start of your shift block, remember). You will also need time to follow the shift turnover procedures, including verifying that the call forwarding to your shift station is working. |
17 | 66 | Christoph Rudolf von rohr | |
18 | 66 | Christoph Rudolf von rohr | * Follow the start-up or quick start-up guide below. |
19 | 66 | Christoph Rudolf von rohr | |
20 | 117 | Ray Neely | * Get in touch with the previous shifter, usually via Slack, and exchange information on the detector status. Follow the [[Shift_turnover]] procedures. |
21 | 66 | Christoph Rudolf von rohr | |
22 | 66 | Christoph Rudolf von rohr | * Exchange information with previous shifters. |
23 | 66 | Christoph Rudolf von rohr | |
24 | 92 | Corey Adams | * Call MCR (use your landline). Inform them that you are off-site, and tell them what station you are at. |
25 | 66 | Christoph Rudolf von rohr | |
26 | 120 | Sophie Berkman | %{background:yellow; color:black}If you encounter problems setting up your system, inform the Run Coordinator. In case of severe connection difficulties such that you cannot fulfill the requirements of shifting, you may also have to inform the backup shifter.% |
27 | 66 | Christoph Rudolf von rohr | |
28 | 66 | Christoph Rudolf von rohr | h3. During your Off-Site Shift |
29 | 66 | Christoph Rudolf von rohr | |
30 | 120 | Sophie Berkman | Although you may be thousands of miles away from the detector, it is very important that the communication between the shifters and experts works out well. Therefore please try to stick to these rules |
31 | 1 | Christoph Rudolf von rohr | |
32 | 120 | Sophie Berkman | * Off-Site shifters must monitor the Slack chat carefully and should use it any time they have questions. |
33 | 99 | Pip Hamilton | |
34 | 120 | Sophie Berkman | * If you need to step out, you can take quick breaks (few minutes) but if you'll be gone for longer arrange for shift coverage by someone who is also shift-trained. Ensure that anyone covering for you knows who to call in case of an emergency (experts and the run coordinator). |
35 | 66 | Christoph Rudolf von rohr | |
36 | 66 | Christoph Rudolf von rohr | h2. Start-up Guide |
37 | 66 | Christoph Rudolf von rohr | |
38 | 120 | Sophie Berkman | This guide lists all steps necessary for starting up your off-site remote shift system. All steps in this guide might change, so if steps in this guide do not work, please look up the information on the "Wiki main page":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki. |
39 | 66 | Christoph Rudolf von rohr | |
40 | 66 | Christoph Rudolf von rohr | h3. Establish Connection to FNAL |
41 | 66 | Christoph Rudolf von rohr | |
42 | 140 | Ralitsa Sharankova | Get a Kerberos Ticket by using your FNAL computing account (e.g. kinit <username>@FNAL.GOV) |
43 | 137 | Sophie Berkman | |
44 | 140 | Ralitsa Sharankova | Please remember to kinit **as yourself** at the start of each remote shift. Coasting on old credentials can cause you to lose connection when they unexpectedly expire. |
45 | 1 | Christoph Rudolf von rohr | |
46 | 121 | Sophie Berkman | %{background:yellow; color:black}It is very important to regularly check that your VNC connections are up to date.% |
47 | 100 | Pip Hamilton | |
48 | 66 | Christoph Rudolf von rohr | h3. Start-up of the Run Control VNC |
49 | 66 | Christoph Rudolf von rohr | |
50 | 66 | Christoph Rudolf von rohr | Before you start up the run control VNC for the first time please check out [[RC - Guide]]. |
51 | 1 | Christoph Rudolf von rohr | |
52 | 100 | Pip Hamilton | Open a terminal/shell **with key forwarding enabled** (option -K) |
53 | 66 | Christoph Rudolf von rohr | <pre> |
54 | 108 | Pip Hamilton | ssh -K -L 5903:localhost:5903 uboonedaq@ubdaq-prod-ws01.fnal.gov |
55 | 66 | Christoph Rudolf von rohr | </pre> |
56 | 66 | Christoph Rudolf von rohr | and on ws01, do |
57 | 1 | Christoph Rudolf von rohr | <pre> |
58 | 1 | Christoph Rudolf von rohr | ssh -L 5903:localhost:5903 uboonedaq@ubdaq-prod-evb.fnal.gov |
59 | 66 | Christoph Rudolf von rohr | </pre> |
60 | 102 | Pip Hamilton | |
61 | 121 | Sophie Berkman | %{background:yellow; color:black} *Do not kinit as yourself* on ws01 (or ws02, or evb). These machines have special principals; if you overwrite them you will lock the DAQ experts out of the system.% |
62 | 1 | Christoph Rudolf von rohr | |
63 | 121 | Sophie Berkman | Now, connect your VNC client to localhost:5903. If you are using RealVNC, open a new terminal and start the VNC session locally on your machine with the command |
64 | 66 | Christoph Rudolf von rohr | <pre> |
65 | 66 | Christoph Rudolf von rohr | vncviewer localhost:5903 |
66 | 66 | Christoph Rudolf von rohr | </pre> |
67 | 110 | Adrien Hourlier | *Use a separate terminal for this command, not in the ws01 or evb window!* |
68 | 1 | Christoph Rudolf von rohr | |
69 | 121 | Sophie Berkman | Enter the standard DAQ password and wait for the VNC to appear. |
70 | 66 | Christoph Rudolf von rohr | |
71 | 66 | Christoph Rudolf von rohr | If you are facing any problems read [[RC - Guide]] and [[RC - Troubleshooting]] before calling the expert. |
72 | 66 | Christoph Rudolf von rohr | |
73 | 1 | Christoph Rudolf von rohr | h3. Start-up the Slow Monitoring/Control VNC |
74 | 66 | Christoph Rudolf von rohr | |
75 | 121 | Sophie Berkman | If you are operating Slow Monitoring/Control for the first time make sure you read [[SLC - Guide]]. |
76 | 66 | Christoph Rudolf von rohr | |
77 | 66 | Christoph Rudolf von rohr | Open a terminal/shell |
78 | 66 | Christoph Rudolf von rohr | <pre> |
79 | 127 | Sophie Berkman | ssh -K -L 5902:localhost:5902 ubooneshift@ubdaq-prod-ws01.fnal.gov |
80 | 66 | Christoph Rudolf von rohr | </pre> |
81 | 121 | Sophie Berkman | now, connect your VNC client to localhost:5902. If you are using RealVNC, open a new terminal and start the VNC session locally on your machine with the command |
82 | 66 | Christoph Rudolf von rohr | <pre> |
83 | 66 | Christoph Rudolf von rohr | vncviewer localhost:5902 |
84 | 66 | Christoph Rudolf von rohr | </pre> |
85 | 141 | Ralitsa Sharankova | Password can be found in the Projects tab on ECL under *SMC common password (VNC viewer)* |
86 | 66 | Christoph Rudolf von rohr | |
87 | 66 | Christoph Rudolf von rohr | If you face any issues while starting up or during operations check [[SLC - Guide]] and [[SLC - Troubleshooting]] before calling the expert. |
88 | 66 | Christoph Rudolf von rohr | |
89 | 1 | Christoph Rudolf von rohr | h3. Start-up PUBS Monitoring GUI |
90 | 66 | Christoph Rudolf von rohr | |
91 | 66 | Christoph Rudolf von rohr | For more information about PUBS read [[DM - Overview]] and [[DM - Shifters]] |
92 | 66 | Christoph Rudolf von rohr | |
93 | 66 | Christoph Rudolf von rohr | Open a terminal/shell |
94 | 66 | Christoph Rudolf von rohr | <pre> |
95 | 140 | Ralitsa Sharankova | ssh -KX uboonedaq@ubdaq-prod-ws01.fnal.gov |
96 | 66 | Christoph Rudolf von rohr | </pre> |
97 | 66 | Christoph Rudolf von rohr | while on ws01 do |
98 | 66 | Christoph Rudolf von rohr | <pre> |
99 | 81 | Christoph Rudolf von rohr | ssh -X uboonedaq@ubdaq-prod-evb.fnal.gov |
100 | 66 | Christoph Rudolf von rohr | </pre> |
101 | 66 | Christoph Rudolf von rohr | on this machine execute the following lines |
102 | 1 | Christoph Rudolf von rohr | <pre> |
103 | 1 | Christoph Rudolf von rohr | cd pubs |
104 | 1 | Christoph Rudolf von rohr | source config/setup.sh |
105 | 1 | Christoph Rudolf von rohr | setup pyqtgraph |
106 | 1 | Christoph Rudolf von rohr | python $PUB_TOP_DIR/pub_mongui/mongui.py |
107 | 1 | Christoph Rudolf von rohr | </pre> |
108 | 1 | Christoph Rudolf von rohr | |
109 | 66 | Christoph Rudolf von rohr | When problems occur follow [[DM - Troubleshooting]] before calling an expert. |
110 | 1 | Christoph Rudolf von rohr | |
111 | 121 | Sophie Berkman | h3. Open a Remote Desktop Session for iFix |
112 | 1 | Christoph Rudolf von rohr | |
113 | 121 | Sophie Berkman | To access iFix, you need to have previously activated your Fermi Domain account and arranged for it to be granted access to the ifix machine by contacting the Run Coordinators and providing your username. |
114 | 1 | Christoph Rudolf von rohr | -Once you have set up the usual VPN connection (above)- |
115 | 121 | Sophie Berkman | |
116 | 82 | Christoph Rudolf von rohr | Connect to ssh tunnel to iFix through ws01: |
117 | 121 | Sophie Berkman | <pre> |
118 | 121 | Sophie Berkman | ssh -C -N -K -L 13389:ppd-ifix2.fnal.gov:3389 ubooneshift@ubdaq-prod-ws01.fnal.gov |
119 | 121 | Sophie Berkman | </pre> |
120 | 86 | Corey Adams | |
121 | 140 | Ralitsa Sharankova | Connect to iFix using a Remote Desktop Protocol client. On Ubuntu, the default is Remmina. For Mac OSX one can use the Microsoft Remote Desktop app, instructions can be found here: https://technet.microsoft.com/en-us/library/dn473012(v=ws.11).aspx |
122 | 121 | Sophie Berkman | |
123 | 1 | Christoph Rudolf von rohr | The information you need for the connection using Remmina is: |
124 | 87 | Corey Adams | * Protocol = RDP (Remote Desktop Protocol) |
125 | 142 | Ralitsa Sharankova | * Server = localhost:13389 |
126 | 121 | Sophie Berkman | * Username = [leave this blank for now] |
127 | 1 | Christoph Rudolf von rohr | * Password = [leave this blank for now] |
128 | 1 | Christoph Rudolf von rohr | * Domain = [leave this blank] |
129 | 1 | Christoph Rudolf von rohr | |
130 | 1 | Christoph Rudolf von rohr | |
131 | 140 | Ralitsa Sharankova | Then, hit connect. A Windows log in screen should load, and you will be prompted to enter your account information: |
132 | 1 | Christoph Rudolf von rohr | * Username = [your own personal Fermi Domain account] |
133 | 1 | Christoph Rudolf von rohr | * Password = [your own personal Fermi Domain password] |
134 | 89 | Corey Adams | |
135 | 1 | Christoph Rudolf von rohr | To connect using Microsoft Remote Desktop use the following: |
136 | 142 | Ralitsa Sharankova | * Server = localhost:13389 |
137 | 140 | Ralitsa Sharankova | * Username = FERMI\< Fermi Domain username> |
138 | 140 | Ralitsa Sharankova | * Password = [leave blank] |
139 | 140 | Ralitsa Sharankova | |
140 | 140 | Ralitsa Sharankova | Then, hit connect. A Windows log screen will appear: |
141 | 140 | Ralitsa Sharankova | * User: select your personal Fermi Domain username |
142 | 140 | Ralitsa Sharankova | * Password = [your personal Fermi Domain password] |
143 | 89 | Corey Adams | |
144 | 91 | Corey Adams | You should see a windows computer load, and ifix will start automatically. Be mindful to only observe the microboone alarm status, and do not make any alterations to any settings in the ifix. When you first load the ifix, you may notice an message: "The alarm queue SUMMARY LIST has overflowed. At least one alarm may have been lost." You can safely dismiss this message, the cryo experts have verified that none of the alarms are lost. In general, if there are alarms right at the start up of the ifix session (red lights, warning messages, etc.) you can safely dismiss them. |
145 | 1 | Christoph Rudolf von rohr | |
146 | 1 | Christoph Rudolf von rohr | h3. Open SSH Tunnel for Web Interfaces |
147 | 86 | Corey Adams | |
148 | 141 | Ralitsa Sharankova | Make sure to setup automatic proxy in your browser settings using this configuration: https://cdcvs.fnal.gov/redmine/attachments/download/24625/lartf.pac |
149 | 141 | Ralitsa Sharankova | |
150 | 84 | Xiao Luo | In order to view the Online Monitoring Lizard, Ganglia and the PUBS status pages one needs to open the following SSH tunnel. This can be done in a separate Terminal by |
151 | 66 | Christoph Rudolf von rohr | <pre> |
152 | 141 | Ralitsa Sharankova | ssh -K -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov |
153 | 66 | Christoph Rudolf von rohr | </pre> |
154 | 66 | Christoph Rudolf von rohr | Now you are ready to start-up the web pages. |
155 | 128 | Sophie Berkman | |
156 | 1 | Christoph Rudolf von rohr | |
157 | 142 | Ralitsa Sharankova | h4. Open Web Pages |
158 | 66 | Christoph Rudolf von rohr | |
159 | 66 | Christoph Rudolf von rohr | The following web pages need to be open during the whole shift. |
160 | 66 | Christoph Rudolf von rohr | |
161 | 66 | Christoph Rudolf von rohr | In the Opera browser open |
162 | 66 | Christoph Rudolf von rohr | |
163 | 66 | Christoph Rudolf von rohr | * "uBooNE Online Monitor Lizard":http://ubdaq-prod-near2.fnal.gov/lizard/ |
164 | 66 | Christoph Rudolf von rohr | |
165 | 109 | Adrien Hourlier | * "MicroBooNE Trigger Page":https://microboone-exp.fnal.gov/at_work/beam/performance/trigger.html |
166 | 66 | Christoph Rudolf von rohr | |
167 | 138 | Glenn Horton-Smith | * "MicroBooNE Run Catalogue":http://ubdaq-prod-near2.fnal.gov/RunCat/web/ |
168 | 138 | Glenn Horton-Smith | |
169 | 138 | Glenn Horton-Smith | * "MicroBooNE Supernova Run Catalog":http://ubdaq-prod-near2.fnal.gov/RunCat/web/recent_sn.cgi |
170 | 96 | David Martinez | |
171 | 80 | Christoph Rudolf von rohr | * "MicroBooNE Run Summary":http://ubdaq-prod-near2.fnal.gov/Pubs/RunSummary.html |
172 | 79 | Christoph Rudolf von rohr | |
173 | 80 | Christoph Rudolf von rohr | * "MicroBooNE Run Statistics":http://ubdaq-prod-near2.fnal.gov/Pubs/RunStat.html |
174 | 1 | Christoph Rudolf von rohr | |
175 | 83 | Christoph Rudolf von rohr | * "Network Weather Map":http://mrtg.fnal.gov/weathermap/wx-uboone.html |
176 | 80 | Christoph Rudolf von rohr | |
177 | 79 | Christoph Rudolf von rohr | |
178 | 74 | Christoph Rudolf von rohr | In any browser of your choice open |
179 | 1 | Christoph Rudolf von rohr | |
180 | 79 | Christoph Rudolf von rohr | * "Ganglia Shifer View":http://ubdaq-prod-evb.fnal.gov:8080/gweb/?r=hour&cs=&ce=&tab=v&vn=shifter |
181 | 1 | Christoph Rudolf von rohr | |
182 | 79 | Christoph Rudolf von rohr | * "MiniBooNE Monitor":http://www-boone.fnal.gov/monitor/ |
183 | 1 | Christoph Rudolf von rohr | |
184 | 79 | Christoph Rudolf von rohr | * "BNB Status Display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/bnbmon/Display |
185 | 1 | Christoph Rudolf von rohr | |
186 | 79 | Christoph Rudolf von rohr | * "NuMi Status Display":http://dbweb6.fnal.gov:8080/ifbeam/bmon/numimon/Display |
187 | 1 | Christoph Rudolf von rohr | |
188 | 79 | Christoph Rudolf von rohr | * "Accelerator Division Notification Page":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top |
189 | 1 | Christoph Rudolf von rohr | |
190 | 126 | Sophie Berkman | * "Slow Monitoring Archiver Status":http://ubdaq-prod-smc.fnal.gov:4812/main |
191 | 66 | Christoph Rudolf von rohr | |
192 | 79 | Christoph Rudolf von rohr | * "MicroBooNE Operations Wiki":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki |
193 | 66 | Christoph Rudolf von rohr | |
194 | 79 | Christoph Rudolf von rohr | * "MicroBooNE Run Plan":https://cdcvs.fnal.gov/redmine/projects/uboone-operations/wiki/Run_plan |
195 | 66 | Christoph Rudolf von rohr | |
196 | 119 | Sophie Berkman | * "MicroBooNE Expert Call List":https://microboone-exp.fnal.gov/at_work/expert_call_list.html |
197 | 79 | Christoph Rudolf von rohr | |
198 | 1 | Christoph Rudolf von rohr | * "Shift Slack Channel":https://microboone.slack.com/messages/CHAD839HQ/ |
199 | 114 | Glenn Horton-Smith | |
200 | 114 | Glenn Horton-Smith | * "MicroBooNE Elog":http://dbweb0.fnal.gov/ECL/uboone/E/index |
201 | 66 | Christoph Rudolf von rohr | |
202 | 66 | Christoph Rudolf von rohr | If you face any problems with the online monitoring Lizard look into [[OM - Checklist]], [[OM - Guide]] and [[OM - Troubleshooting]]. |
203 | 66 | Christoph Rudolf von rohr | |
204 | 66 | Christoph Rudolf von rohr | When you have trouble with the BNB Beam monitor checkout [[Beam - Guide]]. |
205 | 66 | Christoph Rudolf von rohr | |
206 | 66 | Christoph Rudolf von rohr | h3. Communication with ROC West |
207 | 66 | Christoph Rudolf von rohr | |
208 | 94 | Corey Adams | [We do not staff ROC west continuously anymore, so this section is no longer relevant. It's kept here for reference in case we have period where we staff ROC west and a remote center again.] |
209 | 94 | Corey Adams | |
210 | 66 | Christoph Rudolf von rohr | Communication with ROC West is an important part of your off-site shift, therefore: |
211 | 66 | Christoph Rudolf von rohr | |
212 | 78 | Michael Mooney | * Connect to MicroBooNE's CERN Vidyo room using the instructions in [[Off-Site_System_Setup|Off-Site System Setup]]. |
213 | 66 | Christoph Rudolf von rohr | |
214 | 117 | Ray Neely | * Login to the MicroBooNE Slack and join the "shift" channel. |
215 | 66 | Christoph Rudolf von rohr | |
216 | 77 | Michael Mooney | If the Vidyo connection fails, connect to Skype using your local shifting account and call ubooneshift. If both fail, use your landline to communicate with ROC West. Also inform ROC West if you have trouble with your off-site system. |
217 | 66 | Christoph Rudolf von rohr | |
218 | 66 | Christoph Rudolf von rohr | h3. Get ready for Shift |
219 | 66 | Christoph Rudolf von rohr | |
220 | 95 | Corey Adams | If you successfully performed all of the above steps, you are *almost* ready for shift. Call MCR using your landline, give inform them about the off-site shift and leave them your phone number. |
221 | 66 | Christoph Rudolf von rohr | |
222 | 66 | Christoph Rudolf von rohr | Now, you are ready for shift! |
223 | 66 | Christoph Rudolf von rohr | |
224 | 66 | Christoph Rudolf von rohr | h2. Quick Start-up Guide |
225 | 66 | Christoph Rudolf von rohr | |
226 | 139 | Sophie Berkman | -First start the VPN client using your FNAL services account credentials.- |
227 | 112 | Glenn Horton-Smith | |
228 | 112 | Glenn Horton-Smith | Next open a terminal/shell |
229 | 66 | Christoph Rudolf von rohr | <pre> |
230 | 66 | Christoph Rudolf von rohr | ssh -L 5903:localhost:5903 -L 5902:localhost:5902 -D 8881 uboonedaq@ubdaq-prod-ws01.fnal.gov |
231 | 66 | Christoph Rudolf von rohr | </pre> |
232 | 66 | Christoph Rudolf von rohr | and on ws01, do |
233 | 66 | Christoph Rudolf von rohr | <pre> |
234 | 66 | Christoph Rudolf von rohr | ssh -L 5903:localhost:5903 ubdaq-prod-evb.fnal.gov |
235 | 1 | Christoph Rudolf von rohr | </pre> |
236 | 1 | Christoph Rudolf von rohr | |
237 | 112 | Glenn Horton-Smith | Now connect yout VNC client to localhost:5903 and localhost:5902. |
238 | 112 | Glenn Horton-Smith | |
239 | 113 | Glenn Horton-Smith | After that follow the steps to "start up PUBS":/redmine/projects/uboone-operations/wiki/Off-Site_Procedures#Start-up-PUBS-Monitoring-GUI above and open the all the needed pages with your browser(s). |