Wiki » History » Version 260
Monica Nunes, 04/24/2016 03:27 PM
1 | 104 | Jennifer Raaf | !https:/redmine/attachments/download/17373/Lariat_Logo_256.jpg! |
---|---|---|---|
2 | 257 | Jason St. John | |
3 | 257 | Jason St. John | LArIAT shift: *+1 9293 LARIAT* |
4 | 218 | Jason St. John | ROC West x5413 |
5 | 1 | William Badgett | MCenter Control Room (MC-CR) x3726 or x8871 |
6 | 1 | William Badgett | Internationally: prefix *+1 630 840* to the above |
7 | 257 | Jason St. John | |
8 | 241 | Jason St. John | |
9 | 1 | William Badgett | h1. Shifter Stuff |
10 | 1 | William Badgett | |
11 | 147 | Jason St. John | * [[Before your first shift at LArIAT]] |
12 | 254 | Jason St. John | * If remote or starting local shift from scratch: "How to Set up your Shift Station":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/How_to_set_up_your_shift_station |
13 | 249 | Jason St. John | * +[[LArIAT Run 2 Shifter Instructions]]+ (April 2016) |
14 | 207 | Luke Batten | ** [[Run Plan]] |
15 | 202 | Jason St. John | ** "LArIAT Expert Call List":http://lariat.fnal.gov/internal/internal.html (User is "lariat", password same as docdb) |
16 | 248 | Jason St. John | |
17 | 248 | Jason St. John | {{toc}} |
18 | 1 | William Badgett | |
19 | 171 | Jason St. John | h2. Link summary ( [[quick start]] ) |
20 | 166 | Steven Dytman | |
21 | 245 | Jason St. John | * "Live event viewer":http://lariat-wbm.fnal.gov/lariat/evd.html snapshots |
22 | 197 | Jason St. John | * "Accelerator Schedule":http://www-ad.fnal.gov/ops/schedule.html (power outages, etc) |
23 | 246 | Jason St. John | * "LArIAT e-Log":http://dbweb0.fnal.gov/ECL/lariat/E/index For making/ending access, starting/ending shifts. |
24 | 210 | Luke Batten | * [[LArIAT DAQ Instructions]] (February 2016) |
25 | 237 | Jason St. John | * "LArIAT Run Status Webpage":http://lariat-wbm.fnal.gov/lariat/run-alert.html |
26 | 196 | Ryan Linehan | * "LArIAT Run Summary Webpage":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary |
27 | 213 | Luke Batten | * "LArIAT Online DQM":http://lariat-dqm.fnal.gov (works with VPN or VNC or on-site only, for now) |
28 | 164 | Johnny Ho | * How to use the [[event viewers]] |
29 | 1 | William Badgett | * "Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring |
30 | 250 | Jason St. John | * "Beam monitoring (ACNET)":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki |
31 | 247 | Jason St. John | * "Beams Division E-Log":https://www-bd.fnal.gov/Elog/?logNames=Booster%2C+External+Beamlines%2C+Linac%2C+Main+Injector%2C+Operations |
32 | 1 | William Badgett | * "MC7 WebCam":http://131.225.176.26/ |
33 | 1 | William Badgett | |
34 | 151 | Jason St. John | h3. Important knowledge |
35 | 1 | William Badgett | |
36 | 149 | Jason St. John | * "List of MC7 ACNET devices":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/MC7_in_ACNET |
37 | 149 | Jason St. John | * [[MC-7 Enclosure Layout]] |
38 | 1 | William Badgett | * "In case the beam TVs are off":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top |
39 | 1 | William Badgett | * [[The LArIAT supercycle]] _How DAQ operations relate to the main injector supercycle_ |
40 | 151 | Jason St. John | * "Hardware-Connections-and-Configurations":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Hardware-Connections-and-Configurations |
41 | 151 | Jason St. John | * "Electronic Hardware Documentation":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Electronic-Hardware-Documentation |
42 | 132 | Jason St. John | |
43 | 149 | Jason St. John | h3. Obsolete links |
44 | 1 | William Badgett | |
45 | 202 | Jason St. John | * [[LArIAT Run 1 Shifter Instructions]] (June 2015) |
46 | 149 | Jason St. John | * -LArIAT Engineering Run (Aug 2014) [[How to Run the Lariat Data Acquisition]]- (OBSOLETE) |
47 | 1 | William Badgett | * -"Shifter Instructions":https://docs.google.com/document/d/1mlN_UIOCqwhe0cFiU7mRc3-lxDZhY-4TmIy23rOzILI/pub August 2014- (Some obsolete information) |
48 | 164 | Johnny Ho | * -How to make [[Lariat DAQ Plots]]- (OBSOLETE) |
49 | 149 | Jason St. John | |
50 | 83 | Jason St. John | h1. Troubleshooting |
51 | 1 | William Badgett | |
52 | 152 | Jason St. John | Any time day or night, please call experts if you are in need of help and the answer is not provided below. |
53 | 152 | Jason St. John | |
54 | 152 | Jason St. John | h2. *Trigger and DAQ system* |
55 | 152 | Jason St. John | |
56 | 188 | Jason St. John | When the DAQ crashes, please post the relevant information to [[DAQerrors]] for experts to study. |
57 | 188 | Jason St. John | |
58 | 187 | Jason St. John | h3. Common failures: |
59 | 187 | Jason St. John | |
60 | 258 | Jason St. John | * Communication failure with the front ends. Trying clearing it with |
61 | 258 | Jason St. John | <pre>lariatReset</pre> |
62 | 258 | Jason St. John | Wait a few minutes and then try the @go 480@ command again. |
63 | 243 | Jason St. John | * Unable to start a new run: <pre>lost connection |
64 | 243 | Jason St. John | Permission denied (gssapi-keyex,gssapi-with-mic)</pre> |
65 | 259 | Jason St. John | ** Log out of the DAQ machine, and use the command @kticket@ to pick up a fresh kerberos ticket with the special principal that we use for the DAQ: @lariat/lariat@lariat-daq00.fnal.gov@FNAL.GOV@. Check for this success with @klist@. Then log back in to the DAQ machine and start a new run. |
66 | 187 | Jason St. John | * DAQ fails with @ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0@ wait 2 minutes, then try to start again. |
67 | 187 | Jason St. John | ** Try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again. |
68 | 187 | Jason St. John | This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error. |
69 | 187 | Jason St. John | * DAQ fails with @Found persistent communication port still open@ |
70 | 187 | Jason St. John | ** Wait 8 minutes for the ports to clear, then try again. |
71 | 224 | Pawel Kryczynski | * DAQ fails with "An exception occurred when trying to send a message to ..." |
72 | 223 | Pawel Kryczynski | ** Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it). Output from @klist@ should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station. |
73 | 242 | Jason St. John | ** Run number failed increment or has reset to a low number such as 1: Stop the run and then on the DAQ terminal edit @/opt/lariat-online/config/runNumber.dat@ to correct the run number. Start new run after. |
74 | 187 | Jason St. John | |
75 | 187 | Jason St. John | h3. Other failures: |
76 | 187 | Jason St. John | |
77 | 1 | William Badgett | * If the DAQ is having trouble talking to the crates, try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again. |
78 | 187 | Jason St. John | This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error. |
79 | 227 | Jason St. John | * *No triggers at all* If the trigger crate has been power cycled and the V1495 is not emitting triggers: [[Reset1495]] |
80 | 226 | Jason St. John | * *Run Status page not updating* |
81 | 226 | Jason St. John | ** Try refreshing the page. |
82 | 227 | Jason St. John | ** If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: @ssh tomcat@lariat-wbm.fnal.gov@, then @kitty@. (Yep, "kitty.") |
83 | 83 | Jason St. John | |
84 | 152 | Jason St. John | h2. *Wire Chambers* |
85 | 152 | Jason St. John | |
86 | 228 | Jason St. John | Nominal Voltages: |
87 | 229 | Flor de maria Blaszczyk | |_. WC1 | -2425 | |
88 | 228 | Jason St. John | |_. WC2 | -2400 | |
89 | 229 | Flor de maria Blaszczyk | |_. WC3 | -2425 | |
90 | 228 | Jason St. John | |_. WC4 | -2450 | |
91 | 228 | Jason St. John | |
92 | 238 | Flor de maria Blaszczyk | * If the wire chambers HV trips: "instructions for controls via Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#4-Synoptic-power-supplies-controls |
93 | 135 | Jason St. John | * If the wire chambers are acting strangely: "remote power cycling instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Communicating_with_the_wire_chambers including mini-DAQ and telnet interface. |
94 | 83 | Jason St. John | * If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill). The wire chamber controller may still be reading out the hits from the last spill. |
95 | 1 | William Badgett | |
96 | 231 | Flor de maria Blaszczyk | h2. *High voltage for the wire chambers, TPC wireplane bias, cryo PMTs* |
97 | 230 | Flor de maria Blaszczyk | |
98 | 256 | Jason St. John | %{color:red}ICathode HV trips off due to LAr level interlock.% |
99 | 240 | Jason St. John | [[Drift HV]] TPC Cathode Voltage |
100 | 240 | Jason St. John | |
101 | 256 | Jason St. John | "*General Instructions for controls via Synoptic*":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#section-10 |
102 | 230 | Flor de maria Blaszczyk | |
103 | 236 | Flor de maria Blaszczyk | Also valid for some of the muon range stack paddles (1-8), the halo veto and 1 aerogel PMT. |
104 | 236 | Flor de maria Blaszczyk | |
105 | 230 | Flor de maria Blaszczyk | *ONLY IF SYNOPTIC DOESN'T WORK* "Instructions for controls via ACNET":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_to_power_up_the_power_supplies |
106 | 230 | Flor de maria Blaszczyk | |
107 | 230 | Flor de maria Blaszczyk | * The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time. |
108 | 230 | Flor de maria Blaszczyk | This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it. |
109 | 230 | Flor de maria Blaszczyk | |
110 | 230 | Flor de maria Blaszczyk | * The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time) |
111 | 230 | Flor de maria Blaszczyk | This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit. |
112 | 230 | Flor de maria Blaszczyk | |
113 | 230 | Flor de maria Blaszczyk | FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system. |
114 | 230 | Flor de maria Blaszczyk | - Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane. |
115 | 230 | Flor de maria Blaszczyk | - ELT and Hamamatsu PMTs: +2000 V |
116 | 253 | Flor de maria Blaszczyk | - Wire chambers: - 5000 V |
117 | 230 | Flor de maria Blaszczyk | - All of the other units: -2000 V |
118 | 230 | Flor de maria Blaszczyk | |
119 | 230 | Flor de maria Blaszczyk | * Something went wrong during a ramp, I need to turn off the supply! |
120 | 230 | Flor de maria Blaszczyk | There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, *stop the ramp by clicking on the STOP button*. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and *don't forget to turn off the supply!* |
121 | 230 | Flor de maria Blaszczyk | |
122 | 230 | Flor de maria Blaszczyk | *Even if the controls appear not to respond, set the voltage to 0 and turn the supply off*. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken. |
123 | 230 | Flor de maria Blaszczyk | |
124 | 232 | Flor de maria Blaszczyk | * *ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov* |
125 | 232 | Flor de maria Blaszczyk | * *VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov* |
126 | 230 | Flor de maria Blaszczyk | |
127 | 225 | Roberto Acciarri | h2. *Event Viewer* |
128 | 225 | Roberto Acciarri | |
129 | 225 | Roberto Acciarri | * If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like _"event XX in file, but on event 0"_ with _XX_ any number, |
130 | 225 | Roberto Acciarri | ** _*ctrl+z*_ |
131 | 225 | Roberto Acciarri | ** _*ps -def*_ to look at the processes list. Look for the job ID of the process _"python /lariat/app/users/EventViewer/..."_ |
132 | 225 | Roberto Acciarri | ** _*kill -9 JobID*_ to kill the event viewer |
133 | 225 | Roberto Acciarri | ** open a new event viewer |
134 | 225 | Roberto Acciarri | |
135 | 225 | Roberto Acciarri | * If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine |
136 | 181 | Jason St. John | |
137 | 181 | Jason St. John | h2. Power Cycle the ASICs (TPC Wire Noise) |
138 | 181 | Jason St. John | |
139 | 217 | Flor de maria Blaszczyk | * Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be |
140 | 217 | Flor de maria Blaszczyk | alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers |
141 | 219 | Flor de maria Blaszczyk | * You can remotely power cycle the TPC electronics. Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov. Usename "ftbf" (password posted on whiteboard in MC7). Choose the *CYCLE* option, which will automatically give a 60 |
142 | 220 | Flor de maria Blaszczyk | second powerless interlude. |
143 | 220 | Flor de maria Blaszczyk | * Further important informations can be found on the white board in MC-CR. |
144 | 219 | Flor de maria Blaszczyk | * This mechanism is controlled by a smart power distribution unit (PDU) located under the |
145 | 220 | Flor de maria Blaszczyk | TPC electronics power supply near the cryostat in the enclosure. Due to electrical safety |
146 | 220 | Flor de maria Blaszczyk | rules you must *NOT* use any of the other outlets on this PDU. Do *NOT* remove the protective |
147 | 219 | Flor de maria Blaszczyk | outlet covers at any time. |
148 | 220 | Flor de maria Blaszczyk | !https://redmine.fnal.gov/redmine/attachments/download/25467/lariat-pdu02-cycle.png! |
149 | 220 | Flor de maria Blaszczyk | |
150 | 1 | William Badgett | h2. Recover from a LAr level interlock |
151 | 220 | Flor de maria Blaszczyk | |
152 | 221 | Flor de maria Blaszczyk | Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs. |
153 | 221 | Flor de maria Blaszczyk | |
154 | 221 | Flor de maria Blaszczyk | # Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs |
155 | 220 | Flor de maria Blaszczyk | How to control the [[Drift HV]] |
156 | 219 | Flor de maria Blaszczyk | # Power down the ASICs. (They should be off when TPC voltages change.) |
157 | 220 | Flor de maria Blaszczyk | # Bring up the Cathode |
158 | 219 | Flor de maria Blaszczyk | # Bring up the Wire Bias |
159 | 221 | Flor de maria Blaszczyk | # Bring up the ASICs |
160 | 219 | Flor de maria Blaszczyk | # Bring up the PMTs |
161 | 219 | Flor de maria Blaszczyk | |
162 | 219 | Flor de maria Blaszczyk | |
163 | 219 | Flor de maria Blaszczyk | |
164 | 217 | Flor de maria Blaszczyk | h2. Synoptic troubleshooting |
165 | 217 | Flor de maria Blaszczyk | |
166 | 217 | Flor de maria Blaszczyk | %{color:red}"Synoptic Troubleshooting":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#Troubleshooting% |
167 | 217 | Flor de maria Blaszczyk | |
168 | 260 | Monica Nunes | h2. PB S60 PPD Params |
169 | 260 | Monica Nunes | |
170 | 217 | Flor de maria Blaszczyk | |
171 | 68 | Michael Kordosky | h1. Hardware Connections and Configurations |
172 | 1 | William Badgett | |
173 | 141 | William Badgett | * [[TPC Channel Mapping]] |
174 | 51 | Jason St. John | * [[Non-TPC Detectors]] *Naming and numbering in MC7* |
175 | 133 | Jason St. John | * "Cable Patch Panel Spreadsheet":http://goo.gl/JXiH8Y |
176 | 54 | Jason St. John | * [[Connections on the Control Room Patch Panel]] |
177 | 54 | Jason St. John | |
178 | 51 | Jason St. John | * [[Logical inputs to the CAEN V1740]] |
179 | 51 | Jason St. John | * [[Connections into the WUT TDC]] |
180 | 51 | Jason St. John | * [[Connections into the V1751 Digitizer]] |
181 | 1 | William Badgett | * [[Trigger inputs|Connections in/out of the V1495 Trigger]] |
182 | 68 | Michael Kordosky | * [[Logic pulse widths and discriminator thresholds]] |
183 | 68 | Michael Kordosky | * [[WUT configuration]] |
184 | 74 | William Badgett | * [[Timing of ACnet TCLK Events]] |
185 | 140 | William Badgett | * The [[Teensy]] LAr ASIC controller's configuration |
186 | 50 | Michael Kordosky | |
187 | 138 | Roberto Acciarri | * [[Punch Through counters voltages and rates ]] |
188 | 138 | Roberto Acciarri | |
189 | 102 | Jason St. John | h1. Electronic Hardware Documentation |
190 | 45 | Michael Kordosky | |
191 | 51 | Jason St. John | * Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat |
192 | 51 | Jason St. John | * [[LariatOnlineGit]] Accessing the LArIAT Online GIT repository |
193 | 98 | Brian Rebel | * [[Building LArIATOnline with CMake]] |
194 | 44 | Michael Kordosky | |
195 | 51 | Jason St. John | * [[LArIAT TPC Readout Parameters]] |
196 | 185 | William Badgett | * [[LArASIC Preamp Documentation from BNL]] |
197 | 51 | Jason St. John | * [[LArIAT DAQ Commercial Products]] Technical Documentation and Firmware Files |
198 | 101 | William Badgett | * [[LArIAT CAEN Products]] Technical Documentation and Firmware Files |
199 | 51 | Jason St. John | * Test Beam Wire Chamber Sten Hensen's [[TDC Readout Documentation]] |
200 | 89 | William Badgett | * WUT Wave Union TDC: [[Jinyuan's TDC Documentation]] |
201 | 51 | Jason St. John | * [[v1495 notes]] Notes and working area for the V1495 and V2718 |
202 | 195 | Johnny Ho | * [[A3818 PCI optical controller]] links |
203 | 35 | Michael Kordosky | |
204 | 1 | William Badgett | * [[DAQ Configuration]] settings, parameters, starting a run |
205 | 51 | Jason St. John | * [[Raw Data Format and Readout Source code]] |
206 | 100 | John Freeman | * [[All things lariat-artdaq]] (UPDATED 11/5/14) |
207 | 154 | Jason St. John | * [[Interpreting Wire Chamber TDC Data]] |
208 | 36 | Michael Kordosky | |
209 | 54 | Jason St. John | * [[LArIAT Test Stand]] Photographs |
210 | 47 | Michael Kordosky | |
211 | 72 | Michael Kordosky | h1. Diagnostic and Control Software |
212 | 72 | Michael Kordosky | |
213 | 73 | Michael Kordosky | _Test programs in the_ lariat-online/daq/src _directory_ |
214 | 73 | Michael Kordosky | |
215 | 72 | Michael Kordosky | * [[DumpDigitizerData]] |
216 | 72 | Michael Kordosky | * [[SummarizeDAQOutput]] |
217 | 72 | Michael Kordosky | * [[Reset1495]] |
218 | 72 | Michael Kordosky | * [[ScalarReadout]] |
219 | 97 | Pawel Kryczynski | * [[Data_quality_monitoring_module]] |
220 | 72 | Michael Kordosky | |
221 | 139 | William Badgett | * CAEN's [[WaveDump]] utility |
222 | 139 | William Badgett | |
223 | 75 | Jason St. John | h1. Examining the WC mini DAQ ASCII data |
224 | 75 | Jason St. John | |
225 | 75 | Jason St. John | The Processor makes TTrees from the ASCII files saved by @mc7-daq.py@ |
226 | 82 | Jason St. John | [[WC MiniDAQ Data Processor]] |
227 | 75 | Jason St. John | |
228 | 153 | Jason St. John | h1. Data Files from the year 2013 |
229 | 76 | Will Flanagan | |
230 | 85 | William Badgett | Data files for various beam conditions can be found in the [[DataFiles]] page. |
231 | 85 | William Badgett | |
232 | 51 | Jason St. John | h1. Meetings |
233 | 51 | Jason St. John | |
234 | 51 | Jason St. John | * [[Connecting to LArIAT ReadyTalk]] |
235 | 198 | Johnny Ho | |
236 | 1 | William Badgett | h1. Miscellaneous |
237 | 198 | Johnny Ho | |
238 | 216 | Luke Batten | * The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov |
239 | 198 | Johnny Ho | * [[Proxy using SSH tunnel]] |