Project

General

Profile

Wiki » History » Version 304

Andrea Falcone, 05/05/2017 01:26 PM

1 104 Jennifer Raaf
!https:/redmine/attachments/download/17373/Lariat_Logo_256.jpg!  
2 257 Jason St. John
3 272 Jennifer Raaf
LArIAT shift: *+1 9293 LARIAT* (= +1 929 352 7428)
4 218 Jason St. John
ROC West x5413
5 1 William Badgett
MCenter Control Room (MC-CR) x3726 or x8871
6 1 William Badgett
Internationally: prefix *+1 630 840* to the above
7 257 Jason St. John
8 241 Jason St. John
9 1 William Badgett
h1. Shifter Stuff
10 1 William Badgett
11 147 Jason St. John
* [[Before your first shift at LArIAT]]
12 254 Jason St. John
* If remote or starting local shift from scratch: "How to Set up your Shift Station":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/How_to_set_up_your_shift_station
13 279 Andrea Falcone
* +[[LArIAT Run 3 Shifter Instructions]]+ (April 2016)
14 207 Luke Batten
** [[Run Plan]]
15 202 Jason St. John
** "LArIAT Expert Call List":http://lariat.fnal.gov/internal/internal.html  (User is "lariat", password same as docdb)
16 289 Jonathan Asaadi
** "Troubleshooting Page":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki#Troubleshooting
17 248 Jason St. John
18 248 Jason St. John
{{toc}}
19 1 William Badgett
20 171 Jason St. John
h2. Link summary ( [[quick start]] )
21 166 Steven Dytman
22 245 Jason St. John
* "Live event viewer":http://lariat-wbm.fnal.gov/lariat/evd.html snapshots
23 197 Jason St. John
* "Accelerator Schedule":http://www-ad.fnal.gov/ops/schedule.html  (power outages, etc)
24 270 William Badgett
* "LArIAT eLog":http://dbweb0.fnal.gov/ECL/lariat/E/index  For making/ending access, starting/ending shifts.
25 210 Luke Batten
* [[LArIAT DAQ Instructions]] (February 2016)
26 237 Jason St. John
* "LArIAT Run Status Webpage":http://lariat-wbm.fnal.gov/lariat/run-alert.html
27 196 Ryan Linehan
* "LArIAT Run Summary Webpage":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary
28 213 Luke Batten
* "LArIAT Online DQM":http://lariat-dqm.fnal.gov (works with VPN or VNC or on-site only, for now)
29 164 Johnny Ho
* How to use the [[event viewers]]
30 1 William Badgett
* "Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring
31 250 Jason St. John
* "Beam monitoring (ACNET)":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki
32 247 Jason St. John
* "Beams Division E-Log":https://www-bd.fnal.gov/Elog/?logNames=Booster%2C+External+Beamlines%2C+Linac%2C+Main+Injector%2C+Operations
33 1 William Badgett
* "MC7 WebCam":http://131.225.176.26/
34 1 William Badgett
35 151 Jason St. John
h3. Important knowledge
36 1 William Badgett
37 149 Jason St. John
* "List of MC7 ACNET devices":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/MC7_in_ACNET
38 149 Jason St. John
* [[MC-7 Enclosure Layout]]
39 1 William Badgett
* "In case the beam TVs are off":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top
40 1 William Badgett
* [[The LArIAT supercycle]] _How DAQ operations relate to the main injector supercycle_ 
41 151 Jason St. John
* "Hardware-Connections-and-Configurations":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Hardware-Connections-and-Configurations
42 151 Jason St. John
* "Electronic Hardware Documentation":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Electronic-Hardware-Documentation
43 132 Jason St. John
44 149 Jason St. John
h3. Obsolete links
45 1 William Badgett
46 202 Jason St. John
* [[LArIAT Run 1 Shifter Instructions]] (June 2015)
47 149 Jason St. John
* -LArIAT Engineering Run (Aug 2014) [[How to Run the Lariat Data Acquisition]]- (OBSOLETE)
48 1 William Badgett
* -"Shifter Instructions":https://docs.google.com/document/d/1mlN_UIOCqwhe0cFiU7mRc3-lxDZhY-4TmIy23rOzILI/pub August 2014- (Some obsolete information)
49 164 Johnny Ho
* -How to make [[Lariat DAQ Plots]]- (OBSOLETE)
50 149 Jason St. John
51 83 Jason St. John
h1. Troubleshooting 
52 1 William Badgett
53 289 Jonathan Asaadi
%{color:red} *Any time day or night, please call experts if you are in need of help and the answer is not provided below.*%
54 152 Jason St. John
55 274 Jason St. John
h2. *VNC Shared Desktop*
56 274 Jason St. John
57 276 Jason St. John
h3. Mouse clicks not working. Keyboard remapped. Arrow keys type numbers.
58 276 Jason St. John
59 277 Jason St. John
*Solution:* Kill synoptic, restart it.
60 277 Jason St. John
*Procedure:* 
61 277 Jason St. John
* Open a new terminal window on @lariat-cr-02@ (or log in remotely).
62 277 Jason St. John
* Find the process ID's to kill by issuing the command 
63 277 Jason St. John
<pre>ps aux | grep javaws</pre>
64 277 Jason St. John
* The integer after @lariat@ is the process ID.  For example:
65 277 Jason St. John
<pre>
66 277 Jason St. John
[lariat@lariat-cr-02 ~]$ ps aux | grep javaws
67 277 Jason St. John
lariat    1319 19.3 12.2 4049912 471044 pts/1  Sl   Jul15 777:36 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=38857 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawO3BJLo
68 277 Jason St. John
lariat    9599  4.0 12.5 4048312 482184 pts/1  Sl   Jul14 243:28 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=52805 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawBDjN44
69 277 Jason St. John
lariat   13226  0.0  0.0 103312   868 pts/3    S+   10:53   0:00 grep javaws
70 277 Jason St. John
lariat   22629  9.9  7.7 4044088 298244 pts/1  Sl   Jul17  91:52 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=35726 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javaw8bLkDI
71 277 Jason St. John
</pre>
72 277 Jason St. John
* Kill all of these processes: (you may ignore the one for your @grep@) 
73 277 Jason St. John
<pre>kill -9 1319
74 277 Jason St. John
kill -9 9599
75 277 Jason St. John
kill -9 22629</pre>
76 277 Jason St. John
* Check that your keyboard and mouse function are normal again. If not, call an expert.
77 277 Jason St. John
* Restart DAQ or Event Viewer if needed. THEN re-launch Synoptic.
78 276 Jason St. John
79 274 Jason St. John
h3. *'VNC server closed connection'* might mean that the vnc server on lariat-cr-02 is not running. 
80 274 Jason St. John
81 274 Jason St. John
Log on to lariat-cr-02 and check if the X11 vnc sever is running:
82 274 Jason St. John
83 274 Jason St. John
<pre>
84 274 Jason St. John
$ ps aux | grep vnc 
85 274 Jason St. John
lariat 31537 0.0 0.0 103308 860 pts/5 S+ 00:12 0:00 grep vnc 
86 274 Jason St. John
</pre>
87 274 Jason St. John
88 274 Jason St. John
89 275 Jason St. John
Re-launched it:
90 275 Jason St. John
91 282 Jennifer Raaf
[lariat@lariat-cr-02 ~]$ x11vnc -create -localhost -xinerama -display :0 -shared -alwaysshared -many -forever -bg -rfbauth ~/.x11vnc/passwd
92 275 Jason St. John
93 275 Jason St. John
Checked that it is running now:
94 275 Jason St. John
95 275 Jason St. John
[lariat@lariat-cr-02 ~]$ ps aux | grep vnc
96 275 Jason St. John
lariat 31670 8.4 1.6 194032 61944 ? Ss 00:14 0:10 x11vnc -create -localhost -xinerama -display :0 -shared -many -forever -bg -rfbauth /home/lariat/.x11vnc/passwd
97 275 Jason St. John
lariat 31753 0.0 0.0 103308 860 pts/5 S+ 00:16 0:00 grep vnc
98 275 Jason St. John
99 275 Jason St. John
100 275 Jason St. John
...and shifter is able to connect (the most important part!)
101 275 Jason St. John
102 304 Andrea Falcone
103 304 Andrea Falcone
h4. *Keyboard blocked*.
104 304 Andrea Falcone
105 304 Andrea Falcone
First try to press all the control keys (shift, control, alt etc.) and see if this solves the problem.
106 304 Andrea Falcone
107 304 Andrea Falcone
Ask on the hang-out if someone, via VNC, is able to use the keyboard ... sometimes it works.  
108 304 Andrea Falcone
If not, log on to lariat-cr-02, kill the VNC and restart it. 
109 304 Andrea Falcone
110 304 Andrea Falcone
If nothing works someone have to go to ROC-w and hard reboot the PC.
111 304 Andrea Falcone
112 304 Andrea Falcone
113 152 Jason St. John
h2. *Trigger and DAQ system*
114 152 Jason St. John
115 1 William Badgett
When the DAQ crashes, please post the relevant information to [[DAQerrors]] for experts to study.
116 188 Jason St. John
117 289 Jonathan Asaadi
h2. *Common failures:*
118 187 Jason St. John
119 301 Jason St. John
*Communication failure with the front ends.* 
120 301 Jason St. John
* Trying clearing it with 
121 258 Jason St. John
<pre>lariatReset</pre>
122 286 Jason St. John
or 
123 1 William Badgett
<pre>./lariat-online/daq/bin/lariatReset </pre>
124 1 William Badgett
Wait a few minutes and then try the @go 480@ command again.
125 301 Jason St. John
126 301 Jason St. John
*Unable to start a new run:* <pre>lost connection
127 1 William Badgett
Permission denied (gssapi-keyex,gssapi-with-mic)</pre>
128 301 Jason St. John
* Log out of the DAQ machine, and use the command *@kticket@* to pick up a fresh kerberos ticket with the special principal that we use for the DAQ: @lariat/lariat@lariat-daq00.fnal.gov@FNAL.GOV@. Check for this success with @klist@. Then log back in to the DAQ machine and start a new run.
129 301 Jason St. John
130 301 Jason St. John
*Dies with* <pre>ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0</pre>  
131 301 Jason St. John
* Wait 2 minutes, then try to start again.
132 301 Jason St. John
* Try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
133 1 William Badgett
This command is appropriate when one sees any type of "CAEN_DGTZ" error.  This command will NOT help the persistent ports error.
134 301 Jason St. John
135 301 Jason St. John
*DAQ fails with* <pre>Found persistent communication port still open</pre>
136 301 Jason St. John
* Wait 8 minutes for the ports to clear, then try again.
137 301 Jason St. John
138 301 Jason St. John
*DAQ fails with* <pre>An exception occurred when trying to send a message to ..."</pre>
139 301 Jason St. John
* Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it).  Output from @klist@ should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station.
140 301 Jason St. John
141 301 Jason St. John
*Run number fails to increment* or has reset to a low number such as 1: 
142 301 Jason St. John
* Stop the run and then on the DAQ terminal edit @/opt/lariat-online/config/runNumber.dat@ to correct the run number. Start new run after.
143 301 Jason St. John
144 301 Jason St. John
* Unable to start run because <pre>*** Fatal error opening CAEN board at 3,0</pre>
145 301 Jason St. John
146 1 William Badgett
147 288 Andrea Falcone
h3. Run Status page not updating
148 288 Andrea Falcone
149 289 Jonathan Asaadi
* Try refreshing the page. 
150 288 Andrea Falcone
* If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: 
151 227 Jason St. John
** @ssh tomcat@lariat-wbm.fnal.gov@, then @kitty@.  (Yep, "kitty.")
152 226 Jason St. John
* If there are messages of lost connection and failed to copy run.xml.new : 
153 226 Jason St. John
** kill the run (by hitting enter in the DAQ terminal), type the command "kticket" into the DAQ terminal, the start a new run by typing the command "go 500".
154 227 Jason St. John
155 287 Andrea Falcone
h3. Other failures: 
156 83 Jason St. John
157 152 Jason St. John
* If the DAQ is having trouble talking to the crates, try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
158 152 Jason St. John
This command is appropriate when one sees any type of "CAEN_DGTZ" error.  This command will NOT help the persistent ports error.
159 228 Jason St. John
* *No triggers at all* If the trigger crate has been power cycled and the V1495 is not emitting triggers: [[Reset1495]]
160 229 Flor de maria Blaszczyk
161 228 Jason St. John
h2. *Wire Chambers*
162 229 Flor de maria Blaszczyk
163 228 Jason St. John
Nominal Voltages:
164 228 Jason St. John
|_. WC1 | -2425 |
165 285 Jason St. John
|_. WC2 | -2400 |
166 285 Jason St. John
|_. WC3 | -2425 |
167 83 Jason St. John
|_. WC4 | -2450 |
168 1 William Badgett
169 231 Flor de maria Blaszczyk
* If the wire chambers HV trips: *Wire Chamber voltage* "instructions to control wire chamber voltages":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#4-Synoptic-power-supplies-controls
170 230 Flor de maria Blaszczyk
* If the wire chambers are acting strangely: *Wire Chamber Electronics* "remote power cycling instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Communicating_with_the_wire_chambers including mini-DAQ and telnet interface.
171 256 Jason St. John
* If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill).  The wire chamber controller may still be reading out the hits from the last spill.
172 240 Jason St. John
173 240 Jason St. John
h2. *High voltage for the wire chambers, TPC wireplane bias, cryo PMTs*
174 256 Jason St. John
175 230 Flor de maria Blaszczyk
%{color:red}ICathode HV trips off due to LAr level interlock.%
176 236 Flor de maria Blaszczyk
[[Drift HV]] TPC Cathode Voltage
177 236 Flor de maria Blaszczyk
178 230 Flor de maria Blaszczyk
"*General Instructions for controls via Synoptic*":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#section-10
179 230 Flor de maria Blaszczyk
180 230 Flor de maria Blaszczyk
Also valid for some of the muon range stack paddles (1-8), the halo veto and 1 aerogel PMT.
181 230 Flor de maria Blaszczyk
182 230 Flor de maria Blaszczyk
*ONLY IF SYNOPTIC DOESN'T WORK* "Instructions for controls via ACNET":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_to_power_up_the_power_supplies
183 230 Flor de maria Blaszczyk
184 230 Flor de maria Blaszczyk
* The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time.
185 230 Flor de maria Blaszczyk
This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it. 
186 230 Flor de maria Blaszczyk
187 230 Flor de maria Blaszczyk
* The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time)
188 230 Flor de maria Blaszczyk
This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit.
189 253 Flor de maria Blaszczyk
190 230 Flor de maria Blaszczyk
FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system.
191 230 Flor de maria Blaszczyk
- Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane.
192 230 Flor de maria Blaszczyk
- ELT and Hamamatsu PMTs: +2000 V
193 230 Flor de maria Blaszczyk
- Wire chambers: - 5000 V
194 230 Flor de maria Blaszczyk
- All of the other units: -2000 V
195 230 Flor de maria Blaszczyk
196 230 Flor de maria Blaszczyk
* Something went wrong during a ramp, I need to turn off the supply!
197 232 Flor de maria Blaszczyk
There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, *stop the ramp by clicking on the STOP button*. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and *don't forget to turn off the supply!*
198 232 Flor de maria Blaszczyk
199 230 Flor de maria Blaszczyk
*Even if the controls appear not to respond, set the voltage to 0 and turn the supply off*. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken.
200 284 Will Foreman
201 284 Will Foreman
* *ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov*  
202 284 Will Foreman
* *VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov*
203 284 Will Foreman
204 284 Will Foreman
205 284 Will Foreman
h2. *SiPM and ARAPUCA power supply*
206 284 Will Foreman
207 284 Will Foreman
A low-voltage power supply module on the MCenter control room racks is used to power SiPM boards and (as of Run III) the ARAPUCA light detection device.  
208 284 Will Foreman
209 284 Will Foreman
To turn it on:
210 284 Will Foreman
# Switch on the supply
211 284 Will Foreman
# Press "Output on/off" to turn on output
212 284 Will Foreman
# Press "Voltage"
213 284 Will Foreman
# Adjust to 24V using either the knob (left) or the number pad (right) 
214 225 Roberto Acciarri
215 225 Roberto Acciarri
!https://cdcvs.fnal.gov/redmine/attachments/download/39382/arapuca_supply.png!
216 225 Roberto Acciarri
217 225 Roberto Acciarri
218 225 Roberto Acciarri
h2. *Event Viewer*
219 225 Roberto Acciarri
220 268 Jason St. John
* If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like _"event XX in file, but on event 0"_ with _XX_ any number,
221 268 Jason St. John
** _*ctrl+z*_ 
222 225 Roberto Acciarri
** _*ps -def*_ to look at the processes list. Look for the job ID of the process _"python /lariat/app/users/EventViewer/..."_ 
223 268 Jason St. John
** _*kill -9 JobID*_ to kill the event viewer
224 181 Jason St. John
** open a new event viewer <pre>ssh lariat@lariatgpvm01.fnal.gov 
225 181 Jason St. John
cd /lariat/app/users/EventViewer/; source SETUP_ONLINE_EVENTVIEWER.sh</pre>
226 181 Jason St. John
227 217 Flor de maria Blaszczyk
* If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine
228 217 Flor de maria Blaszczyk
229 278 Jason St. John
h2. Power Cycle the ASICs (TPC Wire Noise)
230 220 Flor de maria Blaszczyk
231 219 Flor de maria Blaszczyk
* Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be 
232 220 Flor de maria Blaszczyk
alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers
233 220 Flor de maria Blaszczyk
* You can remotely power cycle the TPC electronics.  Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov.  Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7).  Choose the *CYCLE* option, which will automatically give a 60 second powerless interlude.
234 219 Flor de maria Blaszczyk
* Further important informations can be found on the white board in MC-CR.
235 220 Flor de maria Blaszczyk
* This mechanism is controlled by a smart power distribution unit (PDU) located under the 
236 220 Flor de maria Blaszczyk
TPC electronics power supply near the cryostat in the enclosure.   Due to electrical safety 
237 298 William Badgett
rules you must *NOT* use any of the other outlets on this PDU.  Do *NOT* remove the protective 
238 290 William Badgett
outlet covers at any time.
239 290 William Badgett
!https://redmine.fnal.gov/redmine/attachments/download/25467/lariat-pdu02-cycle.png!
240 300 William Badgett
241 300 William Badgett
h2. Power Cycle the pLAPPD V1742  (V1751 or V1742 readout hang ups)
242 290 William Badgett
243 302 Jason St. John
Error looks like this:
244 302 Jason St. John
<pre>*** Fatal error opening CAEN board at 3,0</pre>
245 302 Jason St. John
246 302 Jason St. John
* Most likely the V1742 has hung up and needs a power cycle.
247 290 William Badgett
* You can remotely power cycle the V1742 digitizers.  On a firefox on lariat-cr-01 or at MCenter, go to http://ftbf-pdu05.fnal.gov.  Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7, or docdb 2187).  Choose the *CYCLE* option, which will automatically give a 30 second powerless interlude.
248 1 William Badgett
* This mechanism is controlled by a smart power distribution unit (PDU) located under pLAPPD readout 
249 220 Flor de maria Blaszczyk
crate near the beam line on the upstream side.
250 221 Flor de maria Blaszczyk
!https://redmine.fnal.gov/redmine/attachments/download/39804/pLAPPD_PDU.png!
251 221 Flor de maria Blaszczyk
252 221 Flor de maria Blaszczyk
h2. Recover from a LAr level interlock
253 220 Flor de maria Blaszczyk
254 219 Flor de maria Blaszczyk
Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs.
255 283 Jennifer Raaf
256 220 Flor de maria Blaszczyk
# Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs
257 219 Flor de maria Blaszczyk
How to control the [[Drift HV]]
258 221 Flor de maria Blaszczyk
# Power down the ASICs. (They should be off when TPC voltages change.)
259 219 Flor de maria Blaszczyk
# Contact one of these people to [[Reset the HV LAr level interlock in IFIX]]
260 219 Flor de maria Blaszczyk
# Bring up the Cathode
261 217 Flor de maria Blaszczyk
# Bring up the Wire Bias
262 217 Flor de maria Blaszczyk
# Bring up the ASICs
263 217 Flor de maria Blaszczyk
# Bring up the PMTs
264 217 Flor de maria Blaszczyk
265 266 Johnny Ho
h2. Synoptic troubleshooting
266 266 Johnny Ho
267 266 Johnny Ho
%{color:red}"Synoptic Troubleshooting":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#Troubleshooting%
268 267 Johnny Ho
269 266 Johnny Ho
h2. DQM troubleshooting
270 266 Johnny Ho
271 267 Johnny Ho
If the DQM is %{color:red}flashing red%...
272 266 Johnny Ho
273 260 Monica Nunes
* Check to see if the DAQ is running. If it isn't running, you should start a new run!
274 260 Monica Nunes
* Check the "LArIAT Run Status page":http://lariat-wbm.fnal.gov/lariat/run-alert.html. If the number under *Complete Files Waiting* is greater than 10, please call a data handling expert!
275 261 Monica Nunes
* Check the disk space on the @/lariat/data@ mount with the command @df -h /lariat/data@ on a @lariatgpvm@ or @lariat-daq@ machine. If the disk space is 100% full, please call a DQM expert!
276 261 Monica Nunes
277 261 Monica Nunes
278 265 Monica Nunes
h2. PB S60 PPD Params
279 217 Flor de maria Blaszczyk
280 68 Michael Kordosky
If F:MC7AN , F:MC7ANB and F:MC7ANP show the values 1  -6 (in red), this is probably a communication problem, where we are not receiving data from them.
281 1 William Badgett
Usually it comes back in a few minutes. 
282 141 William Badgett
If not, call the MCR and ask them if they can see those values.
283 51 Jason St. John
284 281 Jason St. John
h1. Hardware Connections and Configurations
285 54 Jason St. John
286 54 Jason St. John
* [[TPC Channel Mapping]]
287 51 Jason St. John
* [[Non-TPC Detectors]] *Naming and numbering in MC7*
288 51 Jason St. John
* "Cable Patch Panel Spreadsheet":http://goo.gl/JXiH8Y as well as high voltage settings, power supplies, and warm PMT operating curves
289 1 William Badgett
290 68 Michael Kordosky
* [[Logical inputs to the CAEN V1740]]
291 68 Michael Kordosky
* [[Connections into the WUT TDC]]
292 74 William Badgett
* [[Connections into the V1751 Digitizer]]
293 140 William Badgett
* [[Trigger inputs|Connections in/out of the V1495 Trigger]]
294 50 Michael Kordosky
* [[Logic pulse widths and discriminator thresholds]]
295 138 Roberto Acciarri
* [[WUT configuration]]
296 138 Roberto Acciarri
* [[Timing of ACnet TCLK Events]]
297 102 Jason St. John
* The [[Teensy]] LAr ASIC controller's configuration
298 45 Michael Kordosky
299 51 Jason St. John
* [[Punch Through counters voltages and rates ]]
300 51 Jason St. John
301 98 Brian Rebel
h1. Electronic Hardware Documentation
302 44 Michael Kordosky
303 51 Jason St. John
* Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat
304 185 William Badgett
* [[LariatOnlineGit]] Accessing the LArIAT Online GIT repository
305 51 Jason St. John
* [[Building LArIATOnline with CMake]]
306 101 William Badgett
307 51 Jason St. John
* [[LArIAT TPC Readout Parameters]]
308 89 William Badgett
* [[LArASIC Preamp Documentation from BNL]]
309 51 Jason St. John
* [[LArIAT DAQ Commercial Products]] Technical Documentation and Firmware Files
310 273 James Zhu
* [[LArIAT CAEN Products]] Technical Documentation and Firmware Files
311 195 Johnny Ho
* Test Beam Wire Chamber Sten Hensen's [[TDC Readout Documentation]]
312 35 Michael Kordosky
* WUT Wave Union TDC: [[Jinyuan's TDC Documentation]]
313 1 William Badgett
* [[v1495 notes]] Notes and working area for the V1495 and V2718
314 51 Jason St. John
* "V1495 docs on DocDB":http://lartpc-docdb.fnal.gov:8080/cgi-bin/ShowDocument?docid=1698, built from source:fpga/doc/
315 100 John Freeman
* [[A3818 PCI optical controller]] links
316 154 Jason St. John
317 36 Michael Kordosky
* [[DAQ Configuration]] settings, parameters, starting a run
318 54 Jason St. John
* [[Raw Data Format and Readout Source code]]
319 47 Michael Kordosky
* [[All things lariat-artdaq]] (UPDATED 11/5/14)
320 72 Michael Kordosky
* [[Interpreting Wire Chamber TDC Data]]
321 72 Michael Kordosky
322 73 Michael Kordosky
* [[LArIAT Test Stand]] Photographs
323 73 Michael Kordosky
324 72 Michael Kordosky
h1. Diagnostic and Control Software
325 72 Michael Kordosky
326 72 Michael Kordosky
_Test programs in the_ lariat-online/daq/src _directory_
327 72 Michael Kordosky
328 97 Pawel Kryczynski
* [[DumpDigitizerData]]
329 72 Michael Kordosky
* [[SummarizeDAQOutput]]
330 139 William Badgett
* [[Reset1495]]
331 139 William Badgett
* [[ScalarReadout]]
332 75 Jason St. John
* [[Data_quality_monitoring_module]]
333 75 Jason St. John
334 75 Jason St. John
* CAEN's [[WaveDump]] utility
335 82 Jason St. John
336 75 Jason St. John
h1. Examining the WC mini DAQ ASCII data
337 153 Jason St. John
338 76 Will Flanagan
The Processor makes TTrees from the ASCII files saved by @mc7-daq.py@
339 85 William Badgett
[[WC MiniDAQ Data Processor]]
340 85 William Badgett
341 51 Jason St. John
h1. Data Files from the year 2013
342 51 Jason St. John
343 51 Jason St. John
Data files for various beam conditions can be found in the [[DataFiles]] page.
344 198 Johnny Ho
345 1 William Badgett
h1. Meetings
346 198 Johnny Ho
347 216 Luke Batten
* [[Connecting to LArIAT ReadyTalk]]
348 198 Johnny Ho
349 280 Andrea Falcone
h1. Miscellaneous
350 280 Andrea Falcone
351 280 Andrea Falcone
* The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov
352 280 Andrea Falcone
* [[Proxy using SSH tunnel]]
353 280 Andrea Falcone
354 280 Andrea Falcone
355 280 Andrea Falcone
h1. Commissioning tests before Run III
356 1 William Badgett
357 1 William Badgett
* TOF calibration with electrons - Robb Carey
358 1 William Badgett
359 1 William Badgett
If others want to do calibration, please tell me.