Wiki » History » Version 226
Jason St. John, 03/06/2016 11:25 AM
1 | 104 | Jennifer Raaf | !https:/redmine/attachments/download/17373/Lariat_Logo_256.jpg! |
---|---|---|---|
2 | 104 | Jennifer Raaf | |
3 | 218 | Jason St. John | ROC West x5413 |
4 | 1 | William Badgett | MCenter Control Room (MC-CR) x3726 or x8871 |
5 | 117 | Jennifer Raaf | * Call the Main Control Room (x3721) at the start and at the end of shift. This is required, and pleasant. |
6 | 69 | Jason St. John | * If the beam goes away for more than a few spills, look for Accelerator Alarms in ACNET (a default window). If the problem persists a few minutes, and you see nothing about it in the MCR elog, (or you just really want to find out more), call the MCR. They're friendly, though they may be super-busy. |
7 | 25 | William Badgett | |
8 | 150 | Jason St. John | {{toc}} |
9 | 63 | Jason St. John | |
10 | 204 | Luke Batten | |
11 | 147 | Jason St. John | h1. Shifter Stuff |
12 | 1 | William Badgett | |
13 | 207 | Luke Batten | * [[Before your first shift at LArIAT]] |
14 | 202 | Jason St. John | * [[LArIAT Run 2 Shifter Instructions]] (February 2016) |
15 | 130 | Jennifer Raaf | ** [[Run Plan]] |
16 | 192 | Jennifer Raaf | ** "LArIAT Expert Call List":http://lariat.fnal.gov/internal/internal.html (User is "lariat", password same as docdb) |
17 | 1 | William Badgett | |
18 | 171 | Jason St. John | h2. Link summary ( [[quick start]] ) |
19 | 166 | Steven Dytman | |
20 | 197 | Jason St. John | * "Accelerator Schedule":http://www-ad.fnal.gov/ops/schedule.html (power outages, etc) |
21 | 215 | Luke Batten | * "LArIAT e-Log":http://dbweb0.fnal.gov:8080/ECL/lariat/E/index For making/ending access, starting/ending shifts. |
22 | 210 | Luke Batten | * [[LArIAT DAQ Instructions]] (February 2016) |
23 | 147 | Jason St. John | * "LArIAT Run Status Webpage":http://nusoft.fnal.gov/lariat/run.html |
24 | 196 | Ryan Linehan | * "LArIAT Run Summary Webpage":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary |
25 | 213 | Luke Batten | * "LArIAT Online DQM":http://lariat-dqm.fnal.gov (works with VPN or VNC or on-site only, for now) |
26 | 164 | Johnny Ho | * How to use the [[event viewers]] |
27 | 165 | Johnny Ho | * How to make [[DQM plots]] (may be outdated) |
28 | 1 | William Badgett | * "Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring |
29 | 149 | Jason St. John | * "Beam monitoring(ACNET directly)":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki |
30 | 1 | William Badgett | * "Accelerator Operations Elog":https://www-bd.fnal.gov/Elog/?logNames=Operations |
31 | 1 | William Badgett | * "MC7 WebCam":http://131.225.176.26/ |
32 | 1 | William Badgett | |
33 | 151 | Jason St. John | h3. Important knowledge |
34 | 1 | William Badgett | |
35 | 149 | Jason St. John | * "List of MC7 ACNET devices":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/MC7_in_ACNET |
36 | 149 | Jason St. John | * [[MC-7 Enclosure Layout]] |
37 | 1 | William Badgett | * "In case the beam TVs are off":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top |
38 | 1 | William Badgett | * [[The LArIAT supercycle]] _How DAQ operations relate to the main injector supercycle_ |
39 | 151 | Jason St. John | * "Hardware-Connections-and-Configurations":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Hardware-Connections-and-Configurations |
40 | 151 | Jason St. John | * "Electronic Hardware Documentation":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Electronic-Hardware-Documentation |
41 | 132 | Jason St. John | |
42 | 149 | Jason St. John | h3. Obsolete links |
43 | 1 | William Badgett | |
44 | 202 | Jason St. John | * [[LArIAT Run 1 Shifter Instructions]] (June 2015) |
45 | 149 | Jason St. John | * -LArIAT Engineering Run (Aug 2014) [[How to Run the Lariat Data Acquisition]]- (OBSOLETE) |
46 | 1 | William Badgett | * -"Shifter Instructions":https://docs.google.com/document/d/1mlN_UIOCqwhe0cFiU7mRc3-lxDZhY-4TmIy23rOzILI/pub August 2014- (Some obsolete information) |
47 | 164 | Johnny Ho | * -How to make [[Lariat DAQ Plots]]- (OBSOLETE) |
48 | 149 | Jason St. John | |
49 | 83 | Jason St. John | h1. Troubleshooting |
50 | 1 | William Badgett | |
51 | 152 | Jason St. John | Any time day or night, please call experts if you are in need of help and the answer is not provided below. |
52 | 152 | Jason St. John | |
53 | 179 | Jason St. John | h2. *Trouble getting started* |
54 | 152 | Jason St. John | |
55 | 205 | Luke Batten | * If you cannot ssh to FTBFLX machines: Get a kerberos ticket for yourself: @kinit <yourKerberosPrincipal>@, then ssh ftbf_user@ftbflx18 (or ssh ftbf_user@ftbflx15 or ssh ftbf_user@ftbf-cr-03). |
56 | 216 | Luke Batten | * If your Kerberos username and password are good, but you can't log on to FTBFLX[NN] machines: Have someone who CAN log into those machines check that your kerberos principal appears in @~ftbf_user/.k5login@ in the LArIAT section (email the Run Co-ordinator at stjohn@fnal.gov). |
57 | 216 | Luke Batten | * The same ssh errors may also apply to computers at ROC West. Troubleshoot using the above methods. |
58 | 1 | William Badgett | |
59 | 152 | Jason St. John | h2. *Trigger and DAQ system* |
60 | 152 | Jason St. John | |
61 | 188 | Jason St. John | When the DAQ crashes, please post the relevant information to [[DAQerrors]] for experts to study. |
62 | 188 | Jason St. John | |
63 | 187 | Jason St. John | h3. Common failures: |
64 | 187 | Jason St. John | |
65 | 187 | Jason St. John | * DAQ fails with @ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0@ wait 2 minutes, then try to start again. |
66 | 187 | Jason St. John | ** Try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again. |
67 | 187 | Jason St. John | This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error. |
68 | 187 | Jason St. John | * DAQ fails with @Found persistent communication port still open@ |
69 | 187 | Jason St. John | ** Wait 8 minutes for the ports to clear, then try again. |
70 | 224 | Pawel Kryczynski | * DAQ fails with "An exception occurred when trying to send a message to ..." |
71 | 223 | Pawel Kryczynski | ** Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it). Output from @klist@ should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station. |
72 | 187 | Jason St. John | |
73 | 187 | Jason St. John | h3. Other failures: |
74 | 187 | Jason St. John | |
75 | 1 | William Badgett | * If the DAQ is having trouble talking to the crates, try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again. |
76 | 187 | Jason St. John | This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error. |
77 | 187 | Jason St. John | * If the trigger crate has been power cycled and the V1495 is not emitting triggers: [[Reset1495]] |
78 | 226 | Jason St. John | * *Run Status page not updating* |
79 | 226 | Jason St. John | ** Try refreshing the page. |
80 | 226 | Jason St. John | ** If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: @ssh tomcat@lariat-wbm.fnal.gov@, then @kitty@. (Yep, "kitty.") |
81 | 83 | Jason St. John | |
82 | 152 | Jason St. John | h2. *Wire Chambers* |
83 | 152 | Jason St. John | |
84 | 162 | Jason St. John | * If the wire chambers are acting strangely: "remote power cycling instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Communicating_with_the_wire_chambers including mini-DAQ and telnet interface. |
85 | 135 | Jason St. John | * If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill). The wire chamber controller may still be reading out the hits from the last spill. |
86 | 83 | Jason St. John | |
87 | 225 | Roberto Acciarri | |
88 | 225 | Roberto Acciarri | h2. *Event Viewer* |
89 | 225 | Roberto Acciarri | |
90 | 225 | Roberto Acciarri | * If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like _"event XX in file, but on event 0"_ with _XX_ any number, |
91 | 225 | Roberto Acciarri | ** _*ctrl+z*_ |
92 | 225 | Roberto Acciarri | ** _*ps -def*_ to look at the processes list. Look for the job ID of the process _"python /lariat/app/users/EventViewer/..."_ |
93 | 225 | Roberto Acciarri | ** _*kill -9 JobID*_ to kill the event viewer |
94 | 225 | Roberto Acciarri | ** open a new event viewer |
95 | 225 | Roberto Acciarri | |
96 | 225 | Roberto Acciarri | * If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine |
97 | 225 | Roberto Acciarri | |
98 | 172 | Jason St. John | h2. Power Cycle the ASICs (TPC Wire Noise) |
99 | 168 | William Badgett | |
100 | 168 | William Badgett | * Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be |
101 | 169 | William Badgett | alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers |
102 | 203 | Will Foreman | * You can remotely power cycle the TPC electronics. Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov. Usename "ftbf" (password posted on whiteboard in MC7). Choose the *CYCLE* option, which will automatically give a 60 |
103 | 173 | William Badgett | second powerless interlude. |
104 | 173 | William Badgett | * Further important informations can be found on the white board in MC-CR. |
105 | 168 | William Badgett | * This mechanism is controlled by a smart power distribution unit (PDU) located under the |
106 | 168 | William Badgett | TPC electronics power supply near the cryostat in the enclosure. Due to electrical safety |
107 | 170 | William Badgett | rules you must *NOT* use any of the other outlets on this PDU. Do *NOT* remove the protective |
108 | 168 | William Badgett | outlet covers at any time. |
109 | 177 | William Badgett | !https://redmine.fnal.gov/redmine/attachments/download/25467/lariat-pdu02-cycle.png! |
110 | 174 | William Badgett | |
111 | 181 | Jason St. John | h2. Recover from a LAr level interlock |
112 | 181 | Jason St. John | |
113 | 182 | Jason St. John | Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs. |
114 | 182 | Jason St. John | |
115 | 181 | Jason St. John | # Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs |
116 | 183 | Sarah Lockwitz | How to control the [[Drift HV]] |
117 | 181 | Jason St. John | # Power down the ASICs. (They should be off when TPC voltages change.) |
118 | 181 | Jason St. John | # Bring up the Cathode |
119 | 181 | Jason St. John | # Bring up the Wire Bias |
120 | 181 | Jason St. John | # Bring up the ASICs |
121 | 181 | Jason St. John | # Bring up the PMTs |
122 | 181 | Jason St. John | |
123 | 217 | Flor de maria Blaszczyk | h2. VME power supplies known issues (controlled from Synoptic / ACNET) |
124 | 217 | Flor de maria Blaszczyk | |
125 | 219 | Flor de maria Blaszczyk | "Detailed instructions for controls via Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#section-10 |
126 | 220 | Flor de maria Blaszczyk | *ONLY IF THE ABOVE DOESN'T WORK* |
127 | 220 | Flor de maria Blaszczyk | "Instructions for controls via ACNET":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_to_power_up_the_power_supplies |
128 | 219 | Flor de maria Blaszczyk | |
129 | 220 | Flor de maria Blaszczyk | * The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time. |
130 | 220 | Flor de maria Blaszczyk | This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it. |
131 | 219 | Flor de maria Blaszczyk | |
132 | 220 | Flor de maria Blaszczyk | * The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time) |
133 | 220 | Flor de maria Blaszczyk | This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit. |
134 | 1 | William Badgett | |
135 | 220 | Flor de maria Blaszczyk | FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system. |
136 | 221 | Flor de maria Blaszczyk | - Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane. |
137 | 221 | Flor de maria Blaszczyk | - ELT and Hamamatsu PMTs: +2000 V |
138 | 221 | Flor de maria Blaszczyk | - All of the other units: -2000 V |
139 | 220 | Flor de maria Blaszczyk | |
140 | 219 | Flor de maria Blaszczyk | * Something went wrong during a ramp, I need to turn off the supply! |
141 | 220 | Flor de maria Blaszczyk | There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, *stop the ramp by clicking on the STOP button*. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and *don't forget to turn off the supply!* |
142 | 219 | Flor de maria Blaszczyk | |
143 | 221 | Flor de maria Blaszczyk | *Even if the controls appear not to respond, set the voltage to 0 and turn the supply off*. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken. |
144 | 219 | Flor de maria Blaszczyk | |
145 | 219 | Flor de maria Blaszczyk | * ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov |
146 | 219 | Flor de maria Blaszczyk | * VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov |
147 | 219 | Flor de maria Blaszczyk | |
148 | 217 | Flor de maria Blaszczyk | h2. Synoptic troubleshooting |
149 | 217 | Flor de maria Blaszczyk | |
150 | 217 | Flor de maria Blaszczyk | %{color:red}"Synoptic Troubleshooting":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#Troubleshooting% |
151 | 217 | Flor de maria Blaszczyk | |
152 | 217 | Flor de maria Blaszczyk | |
153 | 68 | Michael Kordosky | h1. Hardware Connections and Configurations |
154 | 1 | William Badgett | |
155 | 141 | William Badgett | * [[TPC Channel Mapping]] |
156 | 51 | Jason St. John | * [[Non-TPC Detectors]] *Naming and numbering in MC7* |
157 | 133 | Jason St. John | * "Cable Patch Panel Spreadsheet":http://goo.gl/JXiH8Y |
158 | 54 | Jason St. John | * [[Connections on the Control Room Patch Panel]] |
159 | 54 | Jason St. John | |
160 | 51 | Jason St. John | * [[Logical inputs to the CAEN V1740]] |
161 | 51 | Jason St. John | * [[Connections into the WUT TDC]] |
162 | 51 | Jason St. John | * [[Connections into the V1751 Digitizer]] |
163 | 1 | William Badgett | * [[Trigger inputs|Connections in/out of the V1495 Trigger]] |
164 | 68 | Michael Kordosky | * [[Logic pulse widths and discriminator thresholds]] |
165 | 68 | Michael Kordosky | * [[WUT configuration]] |
166 | 74 | William Badgett | * [[Timing of ACnet TCLK Events]] |
167 | 140 | William Badgett | * The [[Teensy]] LAr ASIC controller's configuration |
168 | 50 | Michael Kordosky | |
169 | 138 | Roberto Acciarri | * [[Punch Through counters voltages and rates ]] |
170 | 138 | Roberto Acciarri | |
171 | 102 | Jason St. John | h1. Electronic Hardware Documentation |
172 | 45 | Michael Kordosky | |
173 | 51 | Jason St. John | * Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat |
174 | 51 | Jason St. John | * [[LariatOnlineGit]] Accessing the LArIAT Online GIT repository |
175 | 98 | Brian Rebel | * [[Building LArIATOnline with CMake]] |
176 | 44 | Michael Kordosky | |
177 | 51 | Jason St. John | * [[LArIAT TPC Readout Parameters]] |
178 | 185 | William Badgett | * [[LArASIC Preamp Documentation from BNL]] |
179 | 51 | Jason St. John | * [[LArIAT DAQ Commercial Products]] Technical Documentation and Firmware Files |
180 | 101 | William Badgett | * [[LArIAT CAEN Products]] Technical Documentation and Firmware Files |
181 | 51 | Jason St. John | * Test Beam Wire Chamber Sten Hensen's [[TDC Readout Documentation]] |
182 | 89 | William Badgett | * WUT Wave Union TDC: [[Jinyuan's TDC Documentation]] |
183 | 51 | Jason St. John | * [[v1495 notes]] Notes and working area for the V1495 and V2718 |
184 | 195 | Johnny Ho | * [[A3818 PCI optical controller]] links |
185 | 35 | Michael Kordosky | |
186 | 1 | William Badgett | * [[DAQ Configuration]] settings, parameters, starting a run |
187 | 51 | Jason St. John | * [[Raw Data Format and Readout Source code]] |
188 | 100 | John Freeman | * [[All things lariat-artdaq]] (UPDATED 11/5/14) |
189 | 154 | Jason St. John | * [[Interpreting Wire Chamber TDC Data]] |
190 | 36 | Michael Kordosky | |
191 | 54 | Jason St. John | * [[LArIAT Test Stand]] Photographs |
192 | 47 | Michael Kordosky | |
193 | 72 | Michael Kordosky | h1. Diagnostic and Control Software |
194 | 72 | Michael Kordosky | |
195 | 73 | Michael Kordosky | _Test programs in the_ lariat-online/daq/src _directory_ |
196 | 73 | Michael Kordosky | |
197 | 72 | Michael Kordosky | * [[DumpDigitizerData]] |
198 | 72 | Michael Kordosky | * [[SummarizeDAQOutput]] |
199 | 72 | Michael Kordosky | * [[Reset1495]] |
200 | 72 | Michael Kordosky | * [[ScalarReadout]] |
201 | 97 | Pawel Kryczynski | * [[Data_quality_monitoring_module]] |
202 | 72 | Michael Kordosky | |
203 | 139 | William Badgett | * CAEN's [[WaveDump]] utility |
204 | 139 | William Badgett | |
205 | 75 | Jason St. John | h1. Examining the WC mini DAQ ASCII data |
206 | 75 | Jason St. John | |
207 | 75 | Jason St. John | The Processor makes TTrees from the ASCII files saved by @mc7-daq.py@ |
208 | 82 | Jason St. John | [[WC MiniDAQ Data Processor]] |
209 | 75 | Jason St. John | |
210 | 153 | Jason St. John | h1. Data Files from the year 2013 |
211 | 76 | Will Flanagan | |
212 | 85 | William Badgett | Data files for various beam conditions can be found in the [[DataFiles]] page. |
213 | 85 | William Badgett | |
214 | 51 | Jason St. John | h1. Meetings |
215 | 51 | Jason St. John | |
216 | 51 | Jason St. John | * [[Connecting to LArIAT ReadyTalk]] |
217 | 198 | Johnny Ho | |
218 | 1 | William Badgett | h1. Miscellaneous |
219 | 198 | Johnny Ho | |
220 | 216 | Luke Batten | * The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov |
221 | 198 | Johnny Ho | * [[Proxy using SSH tunnel]] |