Wiki » History » Version 325
Jason St. John, 12/11/2017 04:25 PM
1 | 104 | Jennifer Raaf | !https:/redmine/attachments/download/17373/Lariat_Logo_256.jpg! |
---|---|---|---|
2 | 257 | Jason St. John | |
3 | 272 | Jennifer Raaf | LArIAT shift: *+1 9293 LARIAT* (= +1 929 352 7428) |
4 | 218 | Jason St. John | ROC West x5413 |
5 | 1 | William Badgett | MCenter Control Room (MC-CR) x3726 or x8871 |
6 | 1 | William Badgett | Internationally: prefix *+1 630 840* to the above |
7 | 257 | Jason St. John | |
8 | 241 | Jason St. John | |
9 | 1 | William Badgett | h1. Shifter Stuff |
10 | 1 | William Badgett | |
11 | 147 | Jason St. John | * [[Before your first shift at LArIAT]] |
12 | 254 | Jason St. John | * If remote or starting local shift from scratch: "How to Set up your Shift Station":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/How_to_set_up_your_shift_station |
13 | 279 | Andrea Falcone | * +[[LArIAT Run 3 Shifter Instructions]]+ (April 2016) |
14 | 207 | Luke Batten | ** [[Run Plan]] |
15 | 202 | Jason St. John | ** "LArIAT Expert Call List":http://lariat.fnal.gov/internal/internal.html (User is "lariat", password same as docdb) |
16 | 289 | Jonathan Asaadi | ** "Troubleshooting Page":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki#Troubleshooting |
17 | 248 | Jason St. John | |
18 | 248 | Jason St. John | {{toc}} |
19 | 1 | William Badgett | |
20 | 171 | Jason St. John | h2. Link summary ( [[quick start]] ) |
21 | 166 | Steven Dytman | |
22 | 245 | Jason St. John | * "Live event viewer":http://lariat-wbm.fnal.gov/lariat/evd.html snapshots |
23 | 197 | Jason St. John | * "Accelerator Schedule":http://www-ad.fnal.gov/ops/schedule.html (power outages, etc) |
24 | 270 | William Badgett | * "LArIAT eLog":http://dbweb0.fnal.gov/ECL/lariat/E/index For making/ending access, starting/ending shifts. |
25 | 210 | Luke Batten | * [[LArIAT DAQ Instructions]] (February 2016) |
26 | 237 | Jason St. John | * "LArIAT Run Status Webpage":http://lariat-wbm.fnal.gov/lariat/run-alert.html |
27 | 196 | Ryan Linehan | * "LArIAT Run Summary Webpage":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary |
28 | 213 | Luke Batten | * "LArIAT Online DQM":http://lariat-dqm.fnal.gov (works with VPN or VNC or on-site only, for now) |
29 | 164 | Johnny Ho | * How to use the [[event viewers]] |
30 | 1 | William Badgett | * "Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring |
31 | 250 | Jason St. John | * "Beam monitoring (ACNET)":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki |
32 | 247 | Jason St. John | * "Beams Division E-Log":https://www-bd.fnal.gov/Elog/?logNames=Booster%2C+External+Beamlines%2C+Linac%2C+Main+Injector%2C+Operations |
33 | 1 | William Badgett | * "MC7 WebCam":http://131.225.176.26/ |
34 | 321 | William Badgett | * How to take a [[Teensy Channel Scan]] run |
35 | 1 | William Badgett | |
36 | 151 | Jason St. John | h3. Important knowledge |
37 | 1 | William Badgett | |
38 | 149 | Jason St. John | * "List of MC7 ACNET devices":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/MC7_in_ACNET |
39 | 149 | Jason St. John | * [[MC-7 Enclosure Layout]] |
40 | 1 | William Badgett | * "In case the beam TVs are off":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top |
41 | 1 | William Badgett | * [[The LArIAT supercycle]] _How DAQ operations relate to the main injector supercycle_ |
42 | 151 | Jason St. John | * "Hardware-Connections-and-Configurations":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Hardware-Connections-and-Configurations |
43 | 151 | Jason St. John | * "Electronic Hardware Documentation":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Electronic-Hardware-Documentation |
44 | 132 | Jason St. John | |
45 | 149 | Jason St. John | h3. Obsolete links |
46 | 1 | William Badgett | |
47 | 202 | Jason St. John | * [[LArIAT Run 1 Shifter Instructions]] (June 2015) |
48 | 149 | Jason St. John | * -LArIAT Engineering Run (Aug 2014) [[How to Run the Lariat Data Acquisition]]- (OBSOLETE) |
49 | 1 | William Badgett | * -"Shifter Instructions":https://docs.google.com/document/d/1mlN_UIOCqwhe0cFiU7mRc3-lxDZhY-4TmIy23rOzILI/pub August 2014- (Some obsolete information) |
50 | 164 | Johnny Ho | * -How to make [[Lariat DAQ Plots]]- (OBSOLETE) |
51 | 149 | Jason St. John | |
52 | 83 | Jason St. John | h1. Troubleshooting |
53 | 1 | William Badgett | |
54 | 289 | Jonathan Asaadi | %{color:red} *Any time day or night, please call experts if you are in need of help and the answer is not provided below.*% |
55 | 152 | Jason St. John | |
56 | 274 | Jason St. John | h2. *VNC Shared Desktop* |
57 | 274 | Jason St. John | |
58 | 276 | Jason St. John | h3. Mouse clicks not working. Keyboard remapped. Arrow keys type numbers. |
59 | 276 | Jason St. John | |
60 | 309 | Jennifer Raaf | *Try this first* |
61 | 306 | Andrea Falcone | |
62 | 309 | Jennifer Raaf | Press the modifier keys one by one (Alt/Option, "Windows", Control or "Apple"/command, Alt/Option, Control) and see if this solves the problem. This has been suggested as a solution by various online forums, for when the keyboard is acting like a modifier key is stuck in VNC. |
63 | 306 | Andrea Falcone | |
64 | 309 | Jennifer Raaf | *Try this second* |
65 | 306 | Andrea Falcone | |
66 | 306 | Andrea Falcone | Kill synoptic, restart it. |
67 | 306 | Andrea Falcone | |
68 | 307 | Andrea Falcone | Procedure: |
69 | 277 | Jason St. John | * Open a new terminal window on @lariat-cr-02@ (or log in remotely). |
70 | 277 | Jason St. John | * Find the process ID's to kill by issuing the command |
71 | 277 | Jason St. John | <pre>ps aux | grep javaws</pre> |
72 | 277 | Jason St. John | * The integer after @lariat@ is the process ID. For example: |
73 | 277 | Jason St. John | <pre> |
74 | 277 | Jason St. John | [lariat@lariat-cr-02 ~]$ ps aux | grep javaws |
75 | 277 | Jason St. John | lariat 1319 19.3 12.2 4049912 471044 pts/1 Sl Jul15 777:36 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=38857 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawO3BJLo |
76 | 277 | Jason St. John | lariat 9599 4.0 12.5 4048312 482184 pts/1 Sl Jul14 243:28 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=52805 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawBDjN44 |
77 | 277 | Jason St. John | lariat 13226 0.0 0.0 103312 868 pts/3 S+ 10:53 0:00 grep javaws |
78 | 277 | Jason St. John | lariat 22629 9.9 7.7 4044088 298244 pts/1 Sl Jul17 91:52 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=35726 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javaw8bLkDI |
79 | 277 | Jason St. John | </pre> |
80 | 277 | Jason St. John | * Kill all of these processes: (you may ignore the one for your @grep@) |
81 | 277 | Jason St. John | <pre>kill -9 1319 |
82 | 1 | William Badgett | kill -9 9599 |
83 | 277 | Jason St. John | kill -9 22629</pre> |
84 | 306 | Andrea Falcone | * Check that your keyboard and mouse function are normal again. |
85 | 277 | Jason St. John | * Restart DAQ or Event Viewer if needed. THEN re-launch Synoptic. |
86 | 276 | Jason St. John | |
87 | 310 | Jennifer Raaf | *Try this third* |
88 | 310 | Jennifer Raaf | |
89 | 310 | Jennifer Raaf | Ask on the hang-out if someone, via VNC, is able to use the keyboard ... sometimes it works. |
90 | 310 | Jennifer Raaf | |
91 | 311 | Jennifer Raaf | *Try this fourth* |
92 | 310 | Jennifer Raaf | |
93 | 310 | Jennifer Raaf | Kill VNC. Restart it. |
94 | 310 | Jennifer Raaf | |
95 | 306 | Andrea Falcone | *'VNC server closed connection'* might mean that the vnc server on lariat-cr-02 is not running. |
96 | 274 | Jason St. John | |
97 | 274 | Jason St. John | Log on to lariat-cr-02 and check if the X11 vnc sever is running: |
98 | 274 | Jason St. John | |
99 | 274 | Jason St. John | <pre> |
100 | 274 | Jason St. John | $ ps aux | grep vnc |
101 | 274 | Jason St. John | lariat 31537 0.0 0.0 103308 860 pts/5 S+ 00:12 0:00 grep vnc |
102 | 274 | Jason St. John | </pre> |
103 | 275 | Jason St. John | |
104 | 1 | William Badgett | |
105 | 310 | Jennifer Raaf | Re-launch it: |
106 | 308 | Andrea Falcone | |
107 | 282 | Jennifer Raaf | <pre> |
108 | 308 | Andrea Falcone | [lariat@lariat-cr-02 ~]$ x11vnc -create -localhost -xinerama -display :0 -shared -alwaysshared -many -forever -bg -rfbauth ~/.x11vnc/passwd |
109 | 1 | William Badgett | </pre> |
110 | 275 | Jason St. John | |
111 | 310 | Jennifer Raaf | Check that it is running now: |
112 | 1 | William Badgett | |
113 | 1 | William Badgett | <pre> |
114 | 306 | Andrea Falcone | [lariat@lariat-cr-02 ~]$ ps aux | grep vnc |
115 | 306 | Andrea Falcone | lariat 31670 8.4 1.6 194032 61944 ? Ss 00:14 0:10 x11vnc -create -localhost -xinerama -display :0 -shared -many -forever -bg -rfbauth /home/lariat/.x11vnc/passwd |
116 | 306 | Andrea Falcone | lariat 31753 0.0 0.0 103308 860 pts/5 S+ 00:16 0:00 grep vnc |
117 | 306 | Andrea Falcone | </pre> |
118 | 306 | Andrea Falcone | |
119 | 306 | Andrea Falcone | ...and shifter is able to connect (the most important part!) |
120 | 306 | Andrea Falcone | |
121 | 306 | Andrea Falcone | |
122 | 306 | Andrea Falcone | *If nothing else works* |
123 | 306 | Andrea Falcone | |
124 | 310 | Jennifer Raaf | Someone has to go to ROC-w and hard reboot the PC... |
125 | 152 | Jason St. John | |
126 | 152 | Jason St. John | h2. *Trigger and DAQ system* |
127 | 1 | William Badgett | |
128 | 188 | Jason St. John | When the DAQ crashes, please post the relevant information to [[DAQerrors]] for experts to study. |
129 | 289 | Jonathan Asaadi | |
130 | 187 | Jason St. John | h2. *Common failures:* |
131 | 301 | Jason St. John | |
132 | 301 | Jason St. John | *Communication failure with the front ends.* |
133 | 258 | Jason St. John | * Trying clearing it with |
134 | 286 | Jason St. John | <pre>lariatReset</pre> |
135 | 1 | William Badgett | or |
136 | 1 | William Badgett | <pre>./lariat-online/daq/bin/lariatReset </pre> |
137 | 322 | Will Foreman | Wait a few minutes and then try the @go 500@ command again. |
138 | 301 | Jason St. John | |
139 | 1 | William Badgett | *Unable to start a new run:* <pre>lost connection |
140 | 301 | Jason St. John | Permission denied (gssapi-keyex,gssapi-with-mic)</pre> |
141 | 301 | Jason St. John | * Log out of the DAQ machine, and use the command *@kticket@* to pick up a fresh kerberos ticket with the special principal that we use for the DAQ: @lariat/lariat@lariat-daq00.fnal.gov@FNAL.GOV@. Check for this success with @klist@. Then log back in to the DAQ machine and start a new run. |
142 | 301 | Jason St. John | |
143 | 301 | Jason St. John | *Dies with* <pre>ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0</pre> |
144 | 301 | Jason St. John | * Wait 2 minutes, then try to start again. |
145 | 1 | William Badgett | * Try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again. |
146 | 301 | Jason St. John | This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error. |
147 | 301 | Jason St. John | |
148 | 301 | Jason St. John | *DAQ fails with* <pre>Found persistent communication port still open</pre> |
149 | 301 | Jason St. John | * Wait 8 minutes for the ports to clear, then try again. |
150 | 301 | Jason St. John | |
151 | 301 | Jason St. John | *DAQ fails with* <pre>An exception occurred when trying to send a message to ..."</pre> |
152 | 301 | Jason St. John | * Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it). Output from @klist@ should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station. |
153 | 301 | Jason St. John | |
154 | 301 | Jason St. John | *Run number fails to increment* or has reset to a low number such as 1: |
155 | 301 | Jason St. John | * Stop the run and then on the DAQ terminal edit @/opt/lariat-online/config/runNumber.dat@ to correct the run number. Start new run after. |
156 | 301 | Jason St. John | |
157 | 288 | Andrea Falcone | |
158 | 288 | Andrea Falcone | h3. Run Status page not updating |
159 | 289 | Jonathan Asaadi | |
160 | 288 | Andrea Falcone | * Try refreshing the page. |
161 | 227 | Jason St. John | * If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: |
162 | 226 | Jason St. John | ** @ssh tomcat@lariat-wbm.fnal.gov@, then @kitty@. (Yep, "kitty.") |
163 | 226 | Jason St. John | * If there are messages of lost connection and failed to copy run.xml.new : |
164 | 227 | Jason St. John | ** kill the run (by hitting enter in the DAQ terminal), type the command "kticket" into the DAQ terminal, the start a new run by typing the command "go 500". |
165 | 287 | Andrea Falcone | |
166 | 312 | William Badgett | h3. LArIAT beam mode not automatically updating Beam / No Beam States |
167 | 312 | William Badgett | |
168 | 312 | William Badgett | * Try restarting the script on an ACnet console, page D129 |
169 | 313 | William Badgett | * Select the *Restart Persistent* option |
170 | 317 | William Badgett | |
171 | 315 | William Badgett | !https://redmine.fnal.gov/redmine/attachments/download/40201/lazarusRising.png! |
172 | 312 | William Badgett | |
173 | 317 | William Badgett | |
174 | 313 | William Badgett | * From list of processes, select *LArIAT_beamCheck.acl* |
175 | 317 | William Badgett | |
176 | 318 | William Badgett | !https://redmine.fnal.gov/redmine/attachments/download/40204/persistenceOfMemory.png! |
177 | 312 | William Badgett | |
178 | 313 | William Badgett | * Confirm by clicking the *Restart* button |
179 | 317 | William Badgett | |
180 | 316 | William Badgett | !https://redmine.fnal.gov/redmine/attachments/download/40203/puddnShang.png! |
181 | 312 | William Badgett | |
182 | 312 | William Badgett | |
183 | 312 | William Badgett | |
184 | 83 | Jason St. John | h3. Other failures: |
185 | 152 | Jason St. John | |
186 | 152 | Jason St. John | * If the DAQ is having trouble talking to the crates, try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again. |
187 | 228 | Jason St. John | This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error. |
188 | 229 | Flor de maria Blaszczyk | * *No triggers at all* If the trigger crate has been power cycled and the V1495 is not emitting triggers: [[Reset1495]] |
189 | 228 | Jason St. John | |
190 | 229 | Flor de maria Blaszczyk | h2. *Wire Chambers* |
191 | 228 | Jason St. John | |
192 | 228 | Jason St. John | Nominal Voltages: |
193 | 285 | Jason St. John | |_. WC1 | -2425 | |
194 | 285 | Jason St. John | |_. WC2 | -2400 | |
195 | 325 | Jason St. John | |_. WC3 | -2550 | |
196 | 1 | William Badgett | |_. WC4 | -2450 | |
197 | 231 | Flor de maria Blaszczyk | |
198 | 230 | Flor de maria Blaszczyk | * If the wire chambers HV trips: *Wire Chamber voltage* "instructions to control wire chamber voltages":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#4-Synoptic-power-supplies-controls |
199 | 256 | Jason St. John | * If the wire chambers are acting strangely: *Wire Chamber Electronics* "remote power cycling instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Communicating_with_the_wire_chambers including mini-DAQ and telnet interface. |
200 | 240 | Jason St. John | * If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill). The wire chamber controller may still be reading out the hits from the last spill. |
201 | 240 | Jason St. John | |
202 | 256 | Jason St. John | h2. *High voltage for the wire chambers, TPC wireplane bias, cryo PMTs* |
203 | 230 | Flor de maria Blaszczyk | |
204 | 236 | Flor de maria Blaszczyk | %{color:red}ICathode HV trips off due to LAr level interlock.% |
205 | 236 | Flor de maria Blaszczyk | [[Drift HV]] TPC Cathode Voltage |
206 | 230 | Flor de maria Blaszczyk | |
207 | 230 | Flor de maria Blaszczyk | "*General Instructions for controls via Synoptic*":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#section-10 |
208 | 230 | Flor de maria Blaszczyk | |
209 | 230 | Flor de maria Blaszczyk | Also valid for some of the muon range stack paddles (1-8), the halo veto and 1 aerogel PMT. |
210 | 230 | Flor de maria Blaszczyk | |
211 | 230 | Flor de maria Blaszczyk | *ONLY IF SYNOPTIC DOESN'T WORK* "Instructions for controls via ACNET":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_to_power_up_the_power_supplies |
212 | 230 | Flor de maria Blaszczyk | |
213 | 230 | Flor de maria Blaszczyk | * The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time. |
214 | 230 | Flor de maria Blaszczyk | This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it. |
215 | 230 | Flor de maria Blaszczyk | |
216 | 230 | Flor de maria Blaszczyk | * The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time) |
217 | 253 | Flor de maria Blaszczyk | This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit. |
218 | 230 | Flor de maria Blaszczyk | |
219 | 230 | Flor de maria Blaszczyk | FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system. |
220 | 230 | Flor de maria Blaszczyk | - Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane. |
221 | 230 | Flor de maria Blaszczyk | - ELT and Hamamatsu PMTs: +2000 V |
222 | 230 | Flor de maria Blaszczyk | - Wire chambers: - 5000 V |
223 | 230 | Flor de maria Blaszczyk | - All of the other units: -2000 V |
224 | 230 | Flor de maria Blaszczyk | |
225 | 232 | Flor de maria Blaszczyk | * Something went wrong during a ramp, I need to turn off the supply! |
226 | 232 | Flor de maria Blaszczyk | There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, *stop the ramp by clicking on the STOP button*. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and *don't forget to turn off the supply!* |
227 | 230 | Flor de maria Blaszczyk | |
228 | 284 | Will Foreman | *Even if the controls appear not to respond, set the voltage to 0 and turn the supply off*. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken. |
229 | 284 | Will Foreman | |
230 | 284 | Will Foreman | * *ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov* |
231 | 284 | Will Foreman | * *VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov* |
232 | 284 | Will Foreman | |
233 | 284 | Will Foreman | |
234 | 284 | Will Foreman | h2. *SiPM and ARAPUCA power supply* |
235 | 284 | Will Foreman | |
236 | 284 | Will Foreman | A low-voltage power supply module on the MCenter control room racks is used to power SiPM boards and (as of Run III) the ARAPUCA light detection device. |
237 | 284 | Will Foreman | |
238 | 284 | Will Foreman | To turn it on: |
239 | 284 | Will Foreman | # Switch on the supply |
240 | 284 | Will Foreman | # Press "Output on/off" to turn on output |
241 | 284 | Will Foreman | # Press "Voltage" |
242 | 225 | Roberto Acciarri | # Adjust to 24V using either the knob (left) or the number pad (right) |
243 | 225 | Roberto Acciarri | |
244 | 225 | Roberto Acciarri | !https://cdcvs.fnal.gov/redmine/attachments/download/39382/arapuca_supply.png! |
245 | 225 | Roberto Acciarri | |
246 | 225 | Roberto Acciarri | |
247 | 225 | Roberto Acciarri | h2. *Event Viewer* |
248 | 268 | Jason St. John | |
249 | 268 | Jason St. John | * If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like _"event XX in file, but on event 0"_ with _XX_ any number, |
250 | 225 | Roberto Acciarri | ** _*ctrl+z*_ |
251 | 268 | Jason St. John | ** _*ps -def*_ to look at the processes list. Look for the job ID of the process _"python /lariat/app/users/EventViewer/..."_ |
252 | 181 | Jason St. John | ** _*kill -9 JobID*_ to kill the event viewer |
253 | 181 | Jason St. John | ** open a new event viewer <pre>ssh lariat@lariatgpvm01.fnal.gov |
254 | 181 | Jason St. John | cd /lariat/app/users/EventViewer/; source SETUP_ONLINE_EVENTVIEWER.sh</pre> |
255 | 217 | Flor de maria Blaszczyk | |
256 | 217 | Flor de maria Blaszczyk | * If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine |
257 | 278 | Jason St. John | |
258 | 220 | Flor de maria Blaszczyk | h2. Power Cycle the ASICs (TPC Wire Noise) |
259 | 219 | Flor de maria Blaszczyk | |
260 | 220 | Flor de maria Blaszczyk | * Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be |
261 | 220 | Flor de maria Blaszczyk | alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers |
262 | 219 | Flor de maria Blaszczyk | * You can remotely power cycle the TPC electronics. Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov. Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7). Choose the *CYCLE* option, which will automatically give a 60 second powerless interlude. |
263 | 220 | Flor de maria Blaszczyk | * Further important informations can be found on the white board in MC-CR. |
264 | 220 | Flor de maria Blaszczyk | * This mechanism is controlled by a smart power distribution unit (PDU) located under the |
265 | 298 | William Badgett | TPC electronics power supply near the cryostat in the enclosure. Due to electrical safety |
266 | 290 | William Badgett | rules you must *NOT* use any of the other outlets on this PDU. Do *NOT* remove the protective |
267 | 290 | William Badgett | outlet covers at any time. |
268 | 300 | William Badgett | !https://redmine.fnal.gov/redmine/attachments/download/25467/lariat-pdu02-cycle.png! |
269 | 300 | William Badgett | |
270 | 290 | William Badgett | h2. Power Cycle the pLAPPD V1742 (V1751 or V1742 readout hang ups) |
271 | 302 | Jason St. John | |
272 | 302 | Jason St. John | Error looks like this: |
273 | 302 | Jason St. John | <pre>*** Fatal error opening CAEN board at 3,0</pre> |
274 | 302 | Jason St. John | |
275 | 290 | William Badgett | * Most likely the V1742 has hung up and needs a power cycle. |
276 | 1 | William Badgett | * You can remotely power cycle the V1742 digitizers. On a firefox on lariat-cr-01 or at MCenter, go to http://ftbf-pdu05.fnal.gov. Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7, or docdb 2187). Choose the *CYCLE* option, which will automatically give a 30 second powerless interlude. |
277 | 220 | Flor de maria Blaszczyk | * This mechanism is controlled by a smart power distribution unit (PDU) located under pLAPPD readout |
278 | 221 | Flor de maria Blaszczyk | crate near the beam line on the upstream side. |
279 | 221 | Flor de maria Blaszczyk | !https://redmine.fnal.gov/redmine/attachments/download/39804/pLAPPD_PDU.png! |
280 | 221 | Flor de maria Blaszczyk | |
281 | 220 | Flor de maria Blaszczyk | h2. Recover from a LAr level interlock |
282 | 219 | Flor de maria Blaszczyk | |
283 | 283 | Jennifer Raaf | Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs. |
284 | 220 | Flor de maria Blaszczyk | |
285 | 219 | Flor de maria Blaszczyk | # Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs |
286 | 221 | Flor de maria Blaszczyk | How to control the [[Drift HV]] |
287 | 219 | Flor de maria Blaszczyk | # Power down the ASICs. (They should be off when TPC voltages change.) |
288 | 219 | Flor de maria Blaszczyk | # Contact one of these people to [[Reset the HV LAr level interlock in IFIX]] |
289 | 217 | Flor de maria Blaszczyk | # Bring up the Cathode |
290 | 217 | Flor de maria Blaszczyk | # Bring up the Wire Bias |
291 | 217 | Flor de maria Blaszczyk | # Bring up the ASICs |
292 | 217 | Flor de maria Blaszczyk | # Bring up the PMTs |
293 | 266 | Johnny Ho | |
294 | 266 | Johnny Ho | h2. Synoptic troubleshooting |
295 | 266 | Johnny Ho | |
296 | 267 | Johnny Ho | %{color:red}"Synoptic Troubleshooting":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#Troubleshooting% |
297 | 266 | Johnny Ho | |
298 | 266 | Johnny Ho | h2. DQM troubleshooting |
299 | 267 | Johnny Ho | |
300 | 266 | Johnny Ho | If the DQM is %{color:red}flashing red%... |
301 | 260 | Monica Nunes | |
302 | 260 | Monica Nunes | * Check to see if the DAQ is running. If it isn't running, you should start a new run! |
303 | 261 | Monica Nunes | * Check the "LArIAT Run Status page":http://lariat-wbm.fnal.gov/lariat/run-alert.html. If the number under *Complete Files Waiting* is greater than 10, please call a data handling expert! |
304 | 261 | Monica Nunes | * Check the disk space on the @/lariat/data@ mount with the command @df -h /lariat/data@ on a @lariatgpvm@ or @lariat-daq@ machine. If the disk space is 100% full, please call a DQM expert! |
305 | 261 | Monica Nunes | |
306 | 323 | Jason St. John | [[Some details of how DQM is set up]] |
307 | 265 | Monica Nunes | |
308 | 217 | Flor de maria Blaszczyk | h2. PB S60 PPD Params |
309 | 68 | Michael Kordosky | |
310 | 1 | William Badgett | If F:MC7AN , F:MC7ANB and F:MC7ANP show the values 1 -6 (in red), this is probably a communication problem, where we are not receiving data from them. |
311 | 141 | William Badgett | Usually it comes back in a few minutes. |
312 | 51 | Jason St. John | If not, call the MCR and ask them if they can see those values. |
313 | 281 | Jason St. John | |
314 | 54 | Jason St. John | h1. Hardware Connections and Configurations |
315 | 54 | Jason St. John | |
316 | 51 | Jason St. John | * [[TPC Channel Mapping]] |
317 | 51 | Jason St. John | * [[Non-TPC Detectors]] *Naming and numbering in MC7* |
318 | 1 | William Badgett | * "Cable Patch Panel Spreadsheet":http://goo.gl/JXiH8Y as well as high voltage settings, power supplies, and warm PMT operating curves |
319 | 68 | Michael Kordosky | |
320 | 68 | Michael Kordosky | * [[Logical inputs to the CAEN V1740]] |
321 | 74 | William Badgett | * [[Connections into the WUT TDC]] |
322 | 140 | William Badgett | * [[Connections into the V1751 Digitizer]] |
323 | 50 | Michael Kordosky | * [[Trigger inputs|Connections in/out of the V1495 Trigger]] |
324 | 138 | Roberto Acciarri | * [[Logic pulse widths and discriminator thresholds]] |
325 | 138 | Roberto Acciarri | * [[WUT configuration]] |
326 | 102 | Jason St. John | * [[Timing of ACnet TCLK Events]] |
327 | 45 | Michael Kordosky | * The [[Teensy]] LAr ASIC controller's configuration |
328 | 51 | Jason St. John | |
329 | 51 | Jason St. John | * [[Punch Through counters voltages and rates ]] |
330 | 98 | Brian Rebel | |
331 | 44 | Michael Kordosky | h1. Electronic Hardware Documentation |
332 | 51 | Jason St. John | |
333 | 185 | William Badgett | * Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat |
334 | 51 | Jason St. John | * [[LariatOnlineGit]] Accessing the LArIAT Online GIT repository |
335 | 101 | William Badgett | * [[Building LArIATOnline with CMake]] |
336 | 51 | Jason St. John | |
337 | 89 | William Badgett | * [[LArIAT TPC Readout Parameters]] |
338 | 51 | Jason St. John | * [[LArASIC Preamp Documentation from BNL]] |
339 | 273 | James Zhu | * [[LArIAT DAQ Commercial Products]] Technical Documentation and Firmware Files |
340 | 195 | Johnny Ho | * [[LArIAT CAEN Products]] Technical Documentation and Firmware Files |
341 | 35 | Michael Kordosky | * Test Beam Wire Chamber Sten Hensen's [[TDC Readout Documentation]] |
342 | 1 | William Badgett | * WUT Wave Union TDC: [[Jinyuan's TDC Documentation]] |
343 | 51 | Jason St. John | * [[v1495 notes]] Notes and working area for the V1495 and V2718 |
344 | 100 | John Freeman | * "V1495 docs on DocDB":http://lartpc-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=1698, built from source:fpga/doc/ |
345 | 154 | Jason St. John | * [[A3818 PCI optical controller]] links |
346 | 36 | Michael Kordosky | |
347 | 54 | Jason St. John | * [[DAQ Configuration]] settings, parameters, starting a run |
348 | 47 | Michael Kordosky | * [[Raw Data Format and Readout Source code]] |
349 | 72 | Michael Kordosky | * [[All things lariat-artdaq]] (UPDATED 11/5/14) |
350 | 72 | Michael Kordosky | * [[Interpreting Wire Chamber TDC Data]] |
351 | 73 | Michael Kordosky | |
352 | 73 | Michael Kordosky | * [[LArIAT Test Stand]] Photographs |
353 | 72 | Michael Kordosky | |
354 | 72 | Michael Kordosky | h1. Diagnostic and Control Software |
355 | 72 | Michael Kordosky | |
356 | 72 | Michael Kordosky | _Test programs in the_ lariat-online/daq/src _directory_ |
357 | 97 | Pawel Kryczynski | |
358 | 72 | Michael Kordosky | * [[DumpDigitizerData]] |
359 | 139 | William Badgett | * [[SummarizeDAQOutput]] |
360 | 139 | William Badgett | * [[Reset1495]] |
361 | 75 | Jason St. John | * [[ScalarReadout]] |
362 | 75 | Jason St. John | * [[Data_quality_monitoring_module]] |
363 | 75 | Jason St. John | |
364 | 82 | Jason St. John | * CAEN's [[WaveDump]] utility |
365 | 75 | Jason St. John | |
366 | 153 | Jason St. John | h1. Examining the WC mini DAQ ASCII data |
367 | 76 | Will Flanagan | |
368 | 85 | William Badgett | The Processor makes TTrees from the ASCII files saved by @mc7-daq.py@ |
369 | 85 | William Badgett | [[WC MiniDAQ Data Processor]] |
370 | 51 | Jason St. John | |
371 | 51 | Jason St. John | h1. Data Files from the year 2013 |
372 | 51 | Jason St. John | |
373 | 198 | Johnny Ho | Data files for various beam conditions can be found in the [[DataFiles]] page. |
374 | 1 | William Badgett | |
375 | 198 | Johnny Ho | h1. Meetings |
376 | 216 | Luke Batten | |
377 | 198 | Johnny Ho | * [[Connecting to LArIAT ReadyTalk]] |
378 | 280 | Andrea Falcone | |
379 | 280 | Andrea Falcone | h1. Miscellaneous |
380 | 280 | Andrea Falcone | |
381 | 280 | Andrea Falcone | * The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov |
382 | 280 | Andrea Falcone | * [[Proxy using SSH tunnel]] |
383 | 280 | Andrea Falcone | |
384 | 1 | William Badgett | h1. Commissioning tests before Run III |
385 | 1 | William Badgett | |
386 | 1 | William Badgett | * TOF calibration with electrons - Robb Carey |
387 | 1 | William Badgett | |
388 | 1 | William Badgett | If others want to do calibration, please tell me. |
389 | 324 | Ron Rechenmacher | |
390 | 324 | Ron Rechenmacher | |
391 | 324 | Ron Rechenmacher | h1. [[November 2017 too many spills per subrun issue]] |