Project

General

Profile

Wiki » History » Version 329

Jonathan Asaadi, 12/29/2017 07:44 PM

1 104 Jennifer Raaf
!https:/redmine/attachments/download/17373/Lariat_Logo_256.jpg!  
2 257 Jason St. John
3 272 Jennifer Raaf
LArIAT shift: *+1 9293 LARIAT* (= +1 929 352 7428)
4 218 Jason St. John
ROC West x5413
5 1 William Badgett
MCenter Control Room (MC-CR) x3726 or x8871
6 1 William Badgett
Internationally: prefix *+1 630 840* to the above
7 257 Jason St. John
8 241 Jason St. John
9 1 William Badgett
h1. Shifter Stuff
10 1 William Badgett
11 147 Jason St. John
* [[Before your first shift at LArIAT]]
12 254 Jason St. John
* If remote or starting local shift from scratch: "How to Set up your Shift Station":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/How_to_set_up_your_shift_station
13 279 Andrea Falcone
* +[[LArIAT Run 3 Shifter Instructions]]+ (April 2016)
14 207 Luke Batten
** [[Run Plan]]
15 202 Jason St. John
** "LArIAT Expert Call List":http://lariat.fnal.gov/internal/internal.html  (User is "lariat", password same as docdb)
16 289 Jonathan Asaadi
** "Troubleshooting Page":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki#Troubleshooting
17 248 Jason St. John
18 248 Jason St. John
{{toc}}
19 1 William Badgett
20 171 Jason St. John
h2. Link summary ( [[quick start]] )
21 166 Steven Dytman
22 245 Jason St. John
* "Live event viewer":http://lariat-wbm.fnal.gov/lariat/evd.html snapshots
23 197 Jason St. John
* "Accelerator Schedule":http://www-ad.fnal.gov/ops/schedule.html  (power outages, etc)
24 270 William Badgett
* "LArIAT eLog":http://dbweb0.fnal.gov/ECL/lariat/E/index  For making/ending access, starting/ending shifts.
25 210 Luke Batten
* [[LArIAT DAQ Instructions]] (February 2016)
26 237 Jason St. John
* "LArIAT Run Status Webpage":http://lariat-wbm.fnal.gov/lariat/run-alert.html
27 196 Ryan Linehan
* "LArIAT Run Summary Webpage":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary
28 213 Luke Batten
* "LArIAT Online DQM":http://lariat-dqm.fnal.gov (works with VPN or VNC or on-site only, for now)
29 164 Johnny Ho
* How to use the [[event viewers]]
30 1 William Badgett
* "Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring
31 250 Jason St. John
* "Beam monitoring (ACNET)":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki
32 247 Jason St. John
* "Beams Division E-Log":https://www-bd.fnal.gov/Elog/?logNames=Booster%2C+External+Beamlines%2C+Linac%2C+Main+Injector%2C+Operations
33 1 William Badgett
* "MC7 WebCam":http://131.225.176.26/
34 321 William Badgett
* How to take a [[Teensy Channel Scan]] run
35 1 William Badgett
36 151 Jason St. John
h3. Important knowledge
37 1 William Badgett
38 149 Jason St. John
* "List of MC7 ACNET devices":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/MC7_in_ACNET
39 149 Jason St. John
* [[MC-7 Enclosure Layout]]
40 1 William Badgett
* "In case the beam TVs are off":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top
41 1 William Badgett
* [[The LArIAT supercycle]] _How DAQ operations relate to the main injector supercycle_ 
42 151 Jason St. John
* "Hardware-Connections-and-Configurations":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Hardware-Connections-and-Configurations
43 151 Jason St. John
* "Electronic Hardware Documentation":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Electronic-Hardware-Documentation
44 132 Jason St. John
45 149 Jason St. John
h3. Obsolete links
46 1 William Badgett
47 202 Jason St. John
* [[LArIAT Run 1 Shifter Instructions]] (June 2015)
48 149 Jason St. John
* -LArIAT Engineering Run (Aug 2014) [[How to Run the Lariat Data Acquisition]]- (OBSOLETE)
49 1 William Badgett
* -"Shifter Instructions":https://docs.google.com/document/d/1mlN_UIOCqwhe0cFiU7mRc3-lxDZhY-4TmIy23rOzILI/pub August 2014- (Some obsolete information)
50 164 Johnny Ho
* -How to make [[Lariat DAQ Plots]]- (OBSOLETE)
51 149 Jason St. John
52 83 Jason St. John
h1. Troubleshooting 
53 1 William Badgett
54 289 Jonathan Asaadi
%{color:red} *Any time day or night, please call experts if you are in need of help and the answer is not provided below.*%
55 152 Jason St. John
56 328 Jonathan Asaadi
h2. *DAQ Dying with errors about Fatal Error Opening to CAEN  board*
57 328 Jonathan Asaadi
58 328 Jonathan Asaadi
If you encounter that the DAQ is dying and can't get it to restart, you may have to reload the firmware on the offending board (as of the writing of this 12/29/17 seems to be board 3).
59 328 Jonathan Asaadi
60 328 Jonathan Asaadi
Very detailed instructions are provided by supreme captain Dr Bill (the amazing) Badgett
61 328 Jonathan Asaadi
62 328 Jonathan Asaadi
"Reload the CAEN Board Firmware":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Reload_CAEN_Board's_Firmware
63 328 Jonathan Asaadi
64 328 Jonathan Asaadi
%{color:green} *Instructions must be followed very carefully and exactly*%
65 328 Jonathan Asaadi
66 328 Jonathan Asaadi
*If you don't know if this is the error that is killing your daq session use the %{color:green}Message Facility MSG Viewer% that pops up when you launch the DAQ and click the top button labelled %{color:green} Error% and you will see if this is your error*
67 328 Jonathan Asaadi
68 329 Jonathan Asaadi
!https://cdcvs.fnal.gov/redmine/attachments/download/45623/Screenshot%20from%202017-12-29%2019-19-22.png!
69 329 Jonathan Asaadi
70 274 Jason St. John
h2. *VNC Shared Desktop*
71 274 Jason St. John
72 276 Jason St. John
h3. Mouse clicks not working. Keyboard remapped. Arrow keys type numbers.
73 276 Jason St. John
74 309 Jennifer Raaf
*Try this first*
75 306 Andrea Falcone
76 309 Jennifer Raaf
Press the modifier keys one by one (Alt/Option, "Windows", Control or "Apple"/command, Alt/Option, Control) and see if this solves the problem. This has been suggested as a solution by various online forums, for when the keyboard is acting like a modifier key is stuck in VNC.
77 306 Andrea Falcone
78 309 Jennifer Raaf
*Try this second* 
79 306 Andrea Falcone
80 306 Andrea Falcone
Kill synoptic, restart it.
81 306 Andrea Falcone
82 307 Andrea Falcone
Procedure: 
83 277 Jason St. John
* Open a new terminal window on @lariat-cr-02@ (or log in remotely).
84 277 Jason St. John
* Find the process ID's to kill by issuing the command 
85 277 Jason St. John
<pre>ps aux | grep javaws</pre>
86 277 Jason St. John
* The integer after @lariat@ is the process ID.  For example:
87 277 Jason St. John
<pre>
88 277 Jason St. John
[lariat@lariat-cr-02 ~]$ ps aux | grep javaws
89 277 Jason St. John
lariat    1319 19.3 12.2 4049912 471044 pts/1  Sl   Jul15 777:36 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=38857 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawO3BJLo
90 277 Jason St. John
lariat    9599  4.0 12.5 4048312 482184 pts/1  Sl   Jul14 243:28 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=52805 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawBDjN44
91 277 Jason St. John
lariat   13226  0.0  0.0 103312   868 pts/3    S+   10:53   0:00 grep javaws
92 277 Jason St. John
lariat   22629  9.9  7.7 4044088 298244 pts/1  Sl   Jul17  91:52 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=35726 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javaw8bLkDI
93 277 Jason St. John
</pre>
94 277 Jason St. John
* Kill all of these processes: (you may ignore the one for your @grep@) 
95 277 Jason St. John
<pre>kill -9 1319
96 1 William Badgett
kill -9 9599
97 277 Jason St. John
kill -9 22629</pre>
98 306 Andrea Falcone
* Check that your keyboard and mouse function are normal again. 
99 277 Jason St. John
* Restart DAQ or Event Viewer if needed. THEN re-launch Synoptic.
100 276 Jason St. John
101 310 Jennifer Raaf
*Try this third*
102 310 Jennifer Raaf
103 310 Jennifer Raaf
Ask on the hang-out if someone, via VNC, is able to use the keyboard ... sometimes it works.  
104 310 Jennifer Raaf
105 311 Jennifer Raaf
*Try this fourth*
106 310 Jennifer Raaf
107 310 Jennifer Raaf
Kill VNC. Restart it.
108 310 Jennifer Raaf
109 306 Andrea Falcone
*'VNC server closed connection'* might mean that the vnc server on lariat-cr-02 is not running. 
110 274 Jason St. John
111 274 Jason St. John
Log on to lariat-cr-02 and check if the X11 vnc sever is running:
112 274 Jason St. John
113 274 Jason St. John
<pre>
114 274 Jason St. John
$ ps aux | grep vnc 
115 274 Jason St. John
lariat 31537 0.0 0.0 103308 860 pts/5 S+ 00:12 0:00 grep vnc 
116 274 Jason St. John
</pre>
117 275 Jason St. John
118 1 William Badgett
119 310 Jennifer Raaf
Re-launch it:
120 308 Andrea Falcone
121 282 Jennifer Raaf
<pre>
122 308 Andrea Falcone
[lariat@lariat-cr-02 ~]$ x11vnc -create -localhost -xinerama -display :0 -shared -alwaysshared -many -forever -bg -rfbauth ~/.x11vnc/passwd
123 1 William Badgett
</pre>
124 275 Jason St. John
125 310 Jennifer Raaf
Check that it is running now:
126 1 William Badgett
127 1 William Badgett
<pre>
128 306 Andrea Falcone
[lariat@lariat-cr-02 ~]$ ps aux | grep vnc
129 306 Andrea Falcone
lariat 31670 8.4 1.6 194032 61944 ? Ss 00:14 0:10 x11vnc -create -localhost -xinerama -display :0 -shared -many -forever -bg -rfbauth /home/lariat/.x11vnc/passwd
130 306 Andrea Falcone
lariat 31753 0.0 0.0 103308 860 pts/5 S+ 00:16 0:00 grep vnc
131 306 Andrea Falcone
</pre>
132 306 Andrea Falcone
133 306 Andrea Falcone
...and shifter is able to connect (the most important part!)
134 306 Andrea Falcone
135 306 Andrea Falcone
136 306 Andrea Falcone
*If nothing else works*
137 306 Andrea Falcone
138 310 Jennifer Raaf
Someone has to go to ROC-w and hard reboot the PC...
139 152 Jason St. John
140 152 Jason St. John
h2. *Trigger and DAQ system*
141 1 William Badgett
142 188 Jason St. John
When the DAQ crashes, please post the relevant information to [[DAQerrors]] for experts to study.
143 289 Jonathan Asaadi
144 187 Jason St. John
h2. *Common failures:*
145 301 Jason St. John
146 301 Jason St. John
*Communication failure with the front ends.* 
147 258 Jason St. John
* Trying clearing it with 
148 286 Jason St. John
<pre>lariatReset</pre>
149 326 William Badgett
150 326 William Badgett
Wait a few seconds and then try the @go 500@ command again.
151 326 William Badgett
152 326 William Badgett
Does the same CAEN digitizer keep giving errors after several lariatReset tries?    If so, you 
153 327 William Badgett
may need to [[Reload CAEN Board's Firmware]]
154 301 Jason St. John
155 1 William Badgett
*Unable to start a new run:* <pre>lost connection
156 301 Jason St. John
Permission denied (gssapi-keyex,gssapi-with-mic)</pre>
157 301 Jason St. John
* Log out of the DAQ machine, and use the command *@kticket@* to pick up a fresh kerberos ticket with the special principal that we use for the DAQ: @lariat/lariat@lariat-daq00.fnal.gov@FNAL.GOV@. Check for this success with @klist@. Then log back in to the DAQ machine and start a new run.
158 301 Jason St. John
159 301 Jason St. John
*Dies with* <pre>ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0</pre>  
160 301 Jason St. John
* Wait 2 minutes, then try to start again.
161 1 William Badgett
* Try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
162 301 Jason St. John
This command is appropriate when one sees any type of "CAEN_DGTZ" error.  This command will NOT help the persistent ports error.
163 301 Jason St. John
164 301 Jason St. John
*DAQ fails with* <pre>Found persistent communication port still open</pre>
165 301 Jason St. John
* Wait 8 minutes for the ports to clear, then try again.
166 301 Jason St. John
167 301 Jason St. John
*DAQ fails with* <pre>An exception occurred when trying to send a message to ..."</pre>
168 301 Jason St. John
* Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it).  Output from @klist@ should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station.
169 301 Jason St. John
170 301 Jason St. John
*Run number fails to increment* or has reset to a low number such as 1: 
171 301 Jason St. John
* Stop the run and then on the DAQ terminal edit @/opt/lariat-online/config/runNumber.dat@ to correct the run number. Start new run after.
172 301 Jason St. John
173 288 Andrea Falcone
174 288 Andrea Falcone
h3. Run Status page not updating
175 289 Jonathan Asaadi
176 288 Andrea Falcone
* Try refreshing the page. 
177 227 Jason St. John
* If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: 
178 226 Jason St. John
** @ssh tomcat@lariat-wbm.fnal.gov@, then @kitty@.  (Yep, "kitty.")
179 226 Jason St. John
* If there are messages of lost connection and failed to copy run.xml.new : 
180 227 Jason St. John
** kill the run (by hitting enter in the DAQ terminal), type the command "kticket" into the DAQ terminal, the start a new run by typing the command "go 500".
181 287 Andrea Falcone
182 312 William Badgett
h3. LArIAT beam mode not automatically updating Beam / No Beam States
183 312 William Badgett
184 312 William Badgett
* Try restarting the script on an ACnet console, page D129
185 313 William Badgett
* Select the *Restart Persistent* option
186 317 William Badgett
187 315 William Badgett
!https://redmine.fnal.gov/redmine/attachments/download/40201/lazarusRising.png!
188 312 William Badgett
189 317 William Badgett
190 313 William Badgett
* From list of processes, select *LArIAT_beamCheck.acl*
191 317 William Badgett
192 318 William Badgett
!https://redmine.fnal.gov/redmine/attachments/download/40204/persistenceOfMemory.png!
193 312 William Badgett
194 313 William Badgett
* Confirm by clicking the *Restart* button
195 317 William Badgett
196 316 William Badgett
!https://redmine.fnal.gov/redmine/attachments/download/40203/puddnShang.png!
197 312 William Badgett
198 312 William Badgett
199 312 William Badgett
200 83 Jason St. John
h3. Other failures: 
201 152 Jason St. John
202 152 Jason St. John
* If the DAQ is having trouble talking to the crates, try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
203 228 Jason St. John
This command is appropriate when one sees any type of "CAEN_DGTZ" error.  This command will NOT help the persistent ports error.
204 229 Flor de maria Blaszczyk
* *No triggers at all* If the trigger crate has been power cycled and the V1495 is not emitting triggers: [[Reset1495]]
205 228 Jason St. John
206 229 Flor de maria Blaszczyk
h2. *Wire Chambers*
207 228 Jason St. John
208 228 Jason St. John
Nominal Voltages:
209 285 Jason St. John
|_. WC1 | -2425 |
210 285 Jason St. John
|_. WC2 | -2400 |
211 325 Jason St. John
|_. WC3 | -2550 |
212 1 William Badgett
|_. WC4 | -2450 |
213 231 Flor de maria Blaszczyk
214 230 Flor de maria Blaszczyk
* If the wire chambers HV trips: *Wire Chamber voltage* "instructions to control wire chamber voltages":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#4-Synoptic-power-supplies-controls
215 256 Jason St. John
* If the wire chambers are acting strangely: *Wire Chamber Electronics* "remote power cycling instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Communicating_with_the_wire_chambers including mini-DAQ and telnet interface.
216 240 Jason St. John
* If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill).  The wire chamber controller may still be reading out the hits from the last spill.
217 240 Jason St. John
218 256 Jason St. John
h2. *High voltage for the wire chambers, TPC wireplane bias, cryo PMTs*
219 230 Flor de maria Blaszczyk
220 236 Flor de maria Blaszczyk
%{color:red}ICathode HV trips off due to LAr level interlock.%
221 236 Flor de maria Blaszczyk
[[Drift HV]] TPC Cathode Voltage
222 230 Flor de maria Blaszczyk
223 230 Flor de maria Blaszczyk
"*General Instructions for controls via Synoptic*":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#section-10
224 230 Flor de maria Blaszczyk
225 230 Flor de maria Blaszczyk
Also valid for some of the muon range stack paddles (1-8), the halo veto and 1 aerogel PMT.
226 230 Flor de maria Blaszczyk
227 230 Flor de maria Blaszczyk
*ONLY IF SYNOPTIC DOESN'T WORK* "Instructions for controls via ACNET":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_to_power_up_the_power_supplies
228 230 Flor de maria Blaszczyk
229 230 Flor de maria Blaszczyk
* The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time.
230 230 Flor de maria Blaszczyk
This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it. 
231 230 Flor de maria Blaszczyk
232 230 Flor de maria Blaszczyk
* The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time)
233 253 Flor de maria Blaszczyk
This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit.
234 230 Flor de maria Blaszczyk
235 230 Flor de maria Blaszczyk
FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system.
236 230 Flor de maria Blaszczyk
- Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane.
237 230 Flor de maria Blaszczyk
- ELT and Hamamatsu PMTs: +2000 V
238 230 Flor de maria Blaszczyk
- Wire chambers: - 5000 V
239 230 Flor de maria Blaszczyk
- All of the other units: -2000 V
240 230 Flor de maria Blaszczyk
241 232 Flor de maria Blaszczyk
* Something went wrong during a ramp, I need to turn off the supply!
242 232 Flor de maria Blaszczyk
There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, *stop the ramp by clicking on the STOP button*. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and *don't forget to turn off the supply!*
243 230 Flor de maria Blaszczyk
244 284 Will Foreman
*Even if the controls appear not to respond, set the voltage to 0 and turn the supply off*. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken.
245 284 Will Foreman
246 284 Will Foreman
* *ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov*  
247 284 Will Foreman
* *VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov*
248 284 Will Foreman
249 284 Will Foreman
250 284 Will Foreman
h2. *SiPM and ARAPUCA power supply*
251 284 Will Foreman
252 284 Will Foreman
A low-voltage power supply module on the MCenter control room racks is used to power SiPM boards and (as of Run III) the ARAPUCA light detection device.  
253 284 Will Foreman
254 284 Will Foreman
To turn it on:
255 284 Will Foreman
# Switch on the supply
256 284 Will Foreman
# Press "Output on/off" to turn on output
257 284 Will Foreman
# Press "Voltage"
258 225 Roberto Acciarri
# Adjust to 24V using either the knob (left) or the number pad (right) 
259 225 Roberto Acciarri
260 225 Roberto Acciarri
!https://cdcvs.fnal.gov/redmine/attachments/download/39382/arapuca_supply.png!
261 225 Roberto Acciarri
262 225 Roberto Acciarri
263 225 Roberto Acciarri
h2. *Event Viewer*
264 268 Jason St. John
265 268 Jason St. John
* If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like _"event XX in file, but on event 0"_ with _XX_ any number,
266 225 Roberto Acciarri
** _*ctrl+z*_ 
267 268 Jason St. John
** _*ps -def*_ to look at the processes list. Look for the job ID of the process _"python /lariat/app/users/EventViewer/..."_ 
268 181 Jason St. John
** _*kill -9 JobID*_ to kill the event viewer
269 181 Jason St. John
** open a new event viewer <pre>ssh lariat@lariatgpvm01.fnal.gov 
270 181 Jason St. John
cd /lariat/app/users/EventViewer/; source SETUP_ONLINE_EVENTVIEWER.sh</pre>
271 217 Flor de maria Blaszczyk
272 217 Flor de maria Blaszczyk
* If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine
273 278 Jason St. John
274 220 Flor de maria Blaszczyk
h2. Power Cycle the ASICs (TPC Wire Noise)
275 219 Flor de maria Blaszczyk
276 220 Flor de maria Blaszczyk
* Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be 
277 220 Flor de maria Blaszczyk
alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers
278 219 Flor de maria Blaszczyk
* You can remotely power cycle the TPC electronics.  Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov.  Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7).  Choose the *CYCLE* option, which will automatically give a 60 second powerless interlude.
279 220 Flor de maria Blaszczyk
* Further important informations can be found on the white board in MC-CR.
280 220 Flor de maria Blaszczyk
* This mechanism is controlled by a smart power distribution unit (PDU) located under the 
281 298 William Badgett
TPC electronics power supply near the cryostat in the enclosure.   Due to electrical safety 
282 290 William Badgett
rules you must *NOT* use any of the other outlets on this PDU.  Do *NOT* remove the protective 
283 290 William Badgett
outlet covers at any time.
284 300 William Badgett
!https://redmine.fnal.gov/redmine/attachments/download/25467/lariat-pdu02-cycle.png!
285 300 William Badgett
286 290 William Badgett
h2. Power Cycle the pLAPPD V1742  (V1751 or V1742 readout hang ups)
287 302 Jason St. John
288 302 Jason St. John
Error looks like this:
289 302 Jason St. John
<pre>*** Fatal error opening CAEN board at 3,0</pre>
290 302 Jason St. John
291 290 William Badgett
* Most likely the V1742 has hung up and needs a power cycle.
292 1 William Badgett
* You can remotely power cycle the V1742 digitizers.  On a firefox on lariat-cr-01 or at MCenter, go to http://ftbf-pdu05.fnal.gov.  Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7, or docdb 2187).  Choose the *CYCLE* option, which will automatically give a 30 second powerless interlude.
293 220 Flor de maria Blaszczyk
* This mechanism is controlled by a smart power distribution unit (PDU) located under pLAPPD readout 
294 221 Flor de maria Blaszczyk
crate near the beam line on the upstream side.
295 221 Flor de maria Blaszczyk
!https://redmine.fnal.gov/redmine/attachments/download/39804/pLAPPD_PDU.png!
296 221 Flor de maria Blaszczyk
297 220 Flor de maria Blaszczyk
h2. Recover from a LAr level interlock
298 219 Flor de maria Blaszczyk
299 283 Jennifer Raaf
Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs.
300 220 Flor de maria Blaszczyk
301 219 Flor de maria Blaszczyk
# Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs
302 221 Flor de maria Blaszczyk
How to control the [[Drift HV]]
303 219 Flor de maria Blaszczyk
# Power down the ASICs. (They should be off when TPC voltages change.)
304 219 Flor de maria Blaszczyk
# Contact one of these people to [[Reset the HV LAr level interlock in IFIX]]
305 217 Flor de maria Blaszczyk
# Bring up the Cathode
306 217 Flor de maria Blaszczyk
# Bring up the Wire Bias
307 217 Flor de maria Blaszczyk
# Bring up the ASICs
308 217 Flor de maria Blaszczyk
# Bring up the PMTs
309 266 Johnny Ho
310 266 Johnny Ho
h2. Synoptic troubleshooting
311 266 Johnny Ho
312 267 Johnny Ho
%{color:red}"Synoptic Troubleshooting":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#Troubleshooting%
313 266 Johnny Ho
314 266 Johnny Ho
h2. DQM troubleshooting
315 267 Johnny Ho
316 266 Johnny Ho
If the DQM is %{color:red}flashing red%...
317 260 Monica Nunes
318 260 Monica Nunes
* Check to see if the DAQ is running. If it isn't running, you should start a new run!
319 261 Monica Nunes
* Check the "LArIAT Run Status page":http://lariat-wbm.fnal.gov/lariat/run-alert.html. If the number under *Complete Files Waiting* is greater than 10, please call a data handling expert!
320 261 Monica Nunes
* Check the disk space on the @/lariat/data@ mount with the command @df -h /lariat/data@ on a @lariatgpvm@ or @lariat-daq@ machine. If the disk space is 100% full, please call a DQM expert!
321 261 Monica Nunes
322 323 Jason St. John
[[Some details of how DQM is set up]]
323 265 Monica Nunes
324 217 Flor de maria Blaszczyk
h2. PB S60 PPD Params
325 68 Michael Kordosky
326 1 William Badgett
If F:MC7AN , F:MC7ANB and F:MC7ANP show the values 1  -6 (in red), this is probably a communication problem, where we are not receiving data from them.
327 141 William Badgett
Usually it comes back in a few minutes. 
328 51 Jason St. John
If not, call the MCR and ask them if they can see those values.
329 281 Jason St. John
330 54 Jason St. John
h1. Hardware Connections and Configurations
331 54 Jason St. John
332 51 Jason St. John
* [[TPC Channel Mapping]]
333 51 Jason St. John
* [[Non-TPC Detectors]] *Naming and numbering in MC7*
334 1 William Badgett
* "Cable Patch Panel Spreadsheet":http://goo.gl/JXiH8Y as well as high voltage settings, power supplies, and warm PMT operating curves
335 68 Michael Kordosky
336 68 Michael Kordosky
* [[Logical inputs to the CAEN V1740]]
337 74 William Badgett
* [[Connections into the WUT TDC]]
338 140 William Badgett
* [[Connections into the V1751 Digitizer]]
339 50 Michael Kordosky
* [[Trigger inputs|Connections in/out of the V1495 Trigger]]
340 138 Roberto Acciarri
* [[Logic pulse widths and discriminator thresholds]]
341 138 Roberto Acciarri
* [[WUT configuration]]
342 102 Jason St. John
* [[Timing of ACnet TCLK Events]]
343 45 Michael Kordosky
* The [[Teensy]] LAr ASIC controller's configuration
344 51 Jason St. John
345 51 Jason St. John
* [[Punch Through counters voltages and rates ]]
346 98 Brian Rebel
347 44 Michael Kordosky
h1. Electronic Hardware Documentation
348 51 Jason St. John
349 185 William Badgett
* Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat
350 51 Jason St. John
* [[LariatOnlineGit]] Accessing the LArIAT Online GIT repository
351 101 William Badgett
* [[Building LArIATOnline with CMake]]
352 51 Jason St. John
353 89 William Badgett
* [[LArIAT TPC Readout Parameters]]
354 51 Jason St. John
* [[LArASIC Preamp Documentation from BNL]]
355 273 James Zhu
* [[LArIAT DAQ Commercial Products]] Technical Documentation and Firmware Files
356 195 Johnny Ho
* [[LArIAT CAEN Products]] Technical Documentation and Firmware Files
357 35 Michael Kordosky
* Test Beam Wire Chamber Sten Hensen's [[TDC Readout Documentation]]
358 1 William Badgett
* WUT Wave Union TDC: [[Jinyuan's TDC Documentation]]
359 51 Jason St. John
* [[v1495 notes]] Notes and working area for the V1495 and V2718
360 100 John Freeman
* "V1495 docs on DocDB":http://lartpc-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=1698, built from source:fpga/doc/
361 154 Jason St. John
* [[A3818 PCI optical controller]] links
362 36 Michael Kordosky
363 54 Jason St. John
* [[DAQ Configuration]] settings, parameters, starting a run
364 47 Michael Kordosky
* [[Raw Data Format and Readout Source code]]
365 72 Michael Kordosky
* [[All things lariat-artdaq]] (UPDATED 11/5/14)
366 72 Michael Kordosky
* [[Interpreting Wire Chamber TDC Data]]
367 73 Michael Kordosky
368 73 Michael Kordosky
* [[LArIAT Test Stand]] Photographs
369 72 Michael Kordosky
370 72 Michael Kordosky
h1. Diagnostic and Control Software
371 72 Michael Kordosky
372 72 Michael Kordosky
_Test programs in the_ lariat-online/daq/src _directory_
373 97 Pawel Kryczynski
374 72 Michael Kordosky
* [[DumpDigitizerData]]
375 139 William Badgett
* [[SummarizeDAQOutput]]
376 139 William Badgett
* [[Reset1495]]
377 75 Jason St. John
* [[ScalarReadout]]
378 75 Jason St. John
* [[Data_quality_monitoring_module]]
379 75 Jason St. John
380 82 Jason St. John
* CAEN's [[WaveDump]] utility
381 75 Jason St. John
382 153 Jason St. John
h1. Examining the WC mini DAQ ASCII data
383 76 Will Flanagan
384 85 William Badgett
The Processor makes TTrees from the ASCII files saved by @mc7-daq.py@
385 85 William Badgett
[[WC MiniDAQ Data Processor]]
386 51 Jason St. John
387 51 Jason St. John
h1. Data Files from the year 2013
388 51 Jason St. John
389 198 Johnny Ho
Data files for various beam conditions can be found in the [[DataFiles]] page.
390 1 William Badgett
391 198 Johnny Ho
h1. Meetings
392 216 Luke Batten
393 198 Johnny Ho
* [[Connecting to LArIAT ReadyTalk]]
394 280 Andrea Falcone
395 280 Andrea Falcone
h1. Miscellaneous
396 280 Andrea Falcone
397 280 Andrea Falcone
* The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov
398 280 Andrea Falcone
* [[Proxy using SSH tunnel]]
399 280 Andrea Falcone
400 1 William Badgett
h1. Commissioning tests before Run III
401 1 William Badgett
402 1 William Badgett
* TOF calibration with electrons - Robb Carey
403 1 William Badgett
404 1 William Badgett
If others want to do calibration, please tell me.
405 324 Ron Rechenmacher
406 324 Ron Rechenmacher
407 324 Ron Rechenmacher
h1. [[November 2017 too many spills per subrun issue]]