Project

General

Profile

Wiki » History » Version 227

Jason St. John, 03/06/2016 11:26 AM

1 104 Jennifer Raaf
!https:/redmine/attachments/download/17373/Lariat_Logo_256.jpg!  
2 104 Jennifer Raaf
 
3 218 Jason St. John
ROC West x5413
4 1 William Badgett
MCenter Control Room (MC-CR) x3726 or x8871
5 117 Jennifer Raaf
* Call the Main Control Room (x3721) at the start and at the end of shift. This is required, and pleasant.
6 69 Jason St. John
* If the beam goes away for more than a few spills, look for Accelerator Alarms in ACNET (a default window). If the problem persists a few minutes, and you see nothing about it in the MCR elog, (or you just really want to find out more), call the MCR.  They're friendly, though they may be super-busy.
7 25 William Badgett
8 150 Jason St. John
{{toc}}
9 63 Jason St. John
10 204 Luke Batten
11 147 Jason St. John
h1. Shifter Stuff
12 1 William Badgett
13 207 Luke Batten
* [[Before your first shift at LArIAT]]
14 202 Jason St. John
* [[LArIAT Run 2 Shifter Instructions]] (February 2016)
15 130 Jennifer Raaf
** [[Run Plan]]
16 192 Jennifer Raaf
** "LArIAT Expert Call List":http://lariat.fnal.gov/internal/internal.html  (User is "lariat", password same as docdb)
17 1 William Badgett
18 171 Jason St. John
h2. Link summary ( [[quick start]] )
19 166 Steven Dytman
20 197 Jason St. John
* "Accelerator Schedule":http://www-ad.fnal.gov/ops/schedule.html  (power outages, etc)
21 215 Luke Batten
* "LArIAT e-Log":http://dbweb0.fnal.gov:8080/ECL/lariat/E/index  For making/ending access, starting/ending shifts.
22 210 Luke Batten
* [[LArIAT DAQ Instructions]] (February 2016)
23 147 Jason St. John
* "LArIAT Run Status Webpage":http://nusoft.fnal.gov/lariat/run.html
24 196 Ryan Linehan
* "LArIAT Run Summary Webpage":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary
25 213 Luke Batten
* "LArIAT Online DQM":http://lariat-dqm.fnal.gov (works with VPN or VNC or on-site only, for now)
26 164 Johnny Ho
* How to use the [[event viewers]]
27 165 Johnny Ho
* How to make [[DQM plots]] (may be outdated)
28 1 William Badgett
* "Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring
29 149 Jason St. John
* "Beam monitoring(ACNET directly)":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki
30 1 William Badgett
* "Accelerator Operations Elog":https://www-bd.fnal.gov/Elog/?logNames=Operations
31 1 William Badgett
* "MC7 WebCam":http://131.225.176.26/
32 1 William Badgett
33 151 Jason St. John
h3. Important knowledge
34 1 William Badgett
35 149 Jason St. John
* "List of MC7 ACNET devices":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/MC7_in_ACNET
36 149 Jason St. John
* [[MC-7 Enclosure Layout]]
37 1 William Badgett
* "In case the beam TVs are off":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top
38 1 William Badgett
* [[The LArIAT supercycle]] _How DAQ operations relate to the main injector supercycle_ 
39 151 Jason St. John
* "Hardware-Connections-and-Configurations":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Hardware-Connections-and-Configurations
40 151 Jason St. John
* "Electronic Hardware Documentation":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Electronic-Hardware-Documentation
41 132 Jason St. John
42 149 Jason St. John
h3. Obsolete links
43 1 William Badgett
44 202 Jason St. John
* [[LArIAT Run 1 Shifter Instructions]] (June 2015)
45 149 Jason St. John
* -LArIAT Engineering Run (Aug 2014) [[How to Run the Lariat Data Acquisition]]- (OBSOLETE)
46 1 William Badgett
* -"Shifter Instructions":https://docs.google.com/document/d/1mlN_UIOCqwhe0cFiU7mRc3-lxDZhY-4TmIy23rOzILI/pub August 2014- (Some obsolete information)
47 164 Johnny Ho
* -How to make [[Lariat DAQ Plots]]- (OBSOLETE)
48 149 Jason St. John
49 83 Jason St. John
h1. Troubleshooting 
50 1 William Badgett
51 152 Jason St. John
Any time day or night, please call experts if you are in need of help and the answer is not provided below.
52 152 Jason St. John
53 179 Jason St. John
h2. *Trouble getting started*
54 152 Jason St. John
55 205 Luke Batten
* If you cannot ssh to FTBFLX machines: Get a kerberos ticket for yourself: @kinit <yourKerberosPrincipal>@, then ssh ftbf_user@ftbflx18 (or ssh ftbf_user@ftbflx15 or ssh ftbf_user@ftbf-cr-03).
56 216 Luke Batten
* If your Kerberos username and password are good, but you can't log on to FTBFLX[NN] machines:  Have someone who CAN log into those machines check that your kerberos principal appears in @~ftbf_user/.k5login@ in the LArIAT section (email the Run Co-ordinator at stjohn@fnal.gov). 
57 216 Luke Batten
* The same ssh errors may also apply to computers at ROC West. Troubleshoot using the above methods.
58 1 William Badgett
59 152 Jason St. John
h2. *Trigger and DAQ system*
60 152 Jason St. John
61 188 Jason St. John
When the DAQ crashes, please post the relevant information to [[DAQerrors]] for experts to study.
62 188 Jason St. John
63 187 Jason St. John
h3. Common failures:
64 187 Jason St. John
65 187 Jason St. John
* DAQ fails with @ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0@  wait 2 minutes, then try to start again.
66 187 Jason St. John
** Try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
67 187 Jason St. John
This command is appropriate when one sees any type of "CAEN_DGTZ" error.  This command will NOT help the persistent ports error.
68 187 Jason St. John
* DAQ fails with @Found persistent communication port still open@
69 187 Jason St. John
** Wait 8 minutes for the ports to clear, then try again.
70 224 Pawel Kryczynski
* DAQ fails with "An exception occurred when trying to send a message to ..." 
71 223 Pawel Kryczynski
** Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it).  Output from @klist@ should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station.
72 187 Jason St. John
73 187 Jason St. John
h3. Other failures: 
74 187 Jason St. John
75 1 William Badgett
* If the DAQ is having trouble talking to the crates, try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
76 187 Jason St. John
This command is appropriate when one sees any type of "CAEN_DGTZ" error.  This command will NOT help the persistent ports error.
77 227 Jason St. John
* *No triggers at all* If the trigger crate has been power cycled and the V1495 is not emitting triggers: [[Reset1495]]
78 226 Jason St. John
* *Run Status page not updating* 
79 226 Jason St. John
** Try refreshing the page. 
80 227 Jason St. John
** If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: @ssh tomcat@lariat-wbm.fnal.gov@, then @kitty@.  (Yep, "kitty.")
81 83 Jason St. John
82 152 Jason St. John
h2. *Wire Chambers*
83 152 Jason St. John
84 162 Jason St. John
* If the wire chambers are acting strangely: "remote power cycling instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Communicating_with_the_wire_chambers including mini-DAQ and telnet interface.
85 135 Jason St. John
* If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill).  The wire chamber controller may still be reading out the hits from the last spill.
86 83 Jason St. John
87 225 Roberto Acciarri
88 225 Roberto Acciarri
h2. *Event Viewer*
89 225 Roberto Acciarri
90 225 Roberto Acciarri
* If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like _"event XX in file, but on event 0"_ with _XX_ any number,
91 225 Roberto Acciarri
** _*ctrl+z*_ 
92 225 Roberto Acciarri
** _*ps -def*_ to look at the processes list. Look for the job ID of the process _"python /lariat/app/users/EventViewer/..."_ 
93 225 Roberto Acciarri
** _*kill -9 JobID*_ to kill the event viewer
94 225 Roberto Acciarri
** open a new event viewer
95 225 Roberto Acciarri
96 225 Roberto Acciarri
* If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine 
97 225 Roberto Acciarri
98 172 Jason St. John
h2. Power Cycle the ASICs (TPC Wire Noise)
99 168 William Badgett
100 168 William Badgett
* Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be 
101 169 William Badgett
alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers
102 203 Will Foreman
* You can remotely power cycle the TPC electronics.  Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov.  Usename "ftbf" (password posted on whiteboard in MC7).  Choose the *CYCLE* option, which will automatically give a 60 
103 173 William Badgett
second powerless interlude.
104 173 William Badgett
* Further important informations can be found on the white board in MC-CR.
105 168 William Badgett
* This mechanism is controlled by a smart power distribution unit (PDU) located under the 
106 168 William Badgett
TPC electronics power supply near the cryostat in the enclosure.   Due to electrical safety 
107 170 William Badgett
rules you must *NOT* use any of the other outlets on this PDU.  Do *NOT* remove the protective 
108 168 William Badgett
outlet covers at any time.
109 177 William Badgett
!https://redmine.fnal.gov/redmine/attachments/download/25467/lariat-pdu02-cycle.png!
110 174 William Badgett
111 181 Jason St. John
h2. Recover from a LAr level interlock
112 181 Jason St. John
113 182 Jason St. John
Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs.
114 182 Jason St. John
115 181 Jason St. John
# Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs
116 183 Sarah Lockwitz
How to control the [[Drift HV]]
117 181 Jason St. John
# Power down the ASICs. (They should be off when TPC voltages change.)
118 181 Jason St. John
# Bring up the Cathode
119 181 Jason St. John
# Bring up the Wire Bias
120 181 Jason St. John
# Bring up the ASICs
121 181 Jason St. John
# Bring up the PMTs
122 181 Jason St. John
123 217 Flor de maria Blaszczyk
h2. VME power supplies known issues (controlled from Synoptic / ACNET)
124 217 Flor de maria Blaszczyk
125 219 Flor de maria Blaszczyk
"Detailed instructions for controls via Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#section-10
126 220 Flor de maria Blaszczyk
*ONLY IF THE ABOVE DOESN'T WORK* 
127 220 Flor de maria Blaszczyk
"Instructions for controls via ACNET":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_to_power_up_the_power_supplies
128 219 Flor de maria Blaszczyk
129 220 Flor de maria Blaszczyk
* The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time.
130 220 Flor de maria Blaszczyk
This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it. 
131 219 Flor de maria Blaszczyk
132 220 Flor de maria Blaszczyk
* The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time)
133 220 Flor de maria Blaszczyk
This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit.
134 1 William Badgett
135 220 Flor de maria Blaszczyk
FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system.
136 221 Flor de maria Blaszczyk
- Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane.
137 221 Flor de maria Blaszczyk
- ELT and Hamamatsu PMTs: +2000 V
138 221 Flor de maria Blaszczyk
- All of the other units: -2000 V
139 220 Flor de maria Blaszczyk
140 219 Flor de maria Blaszczyk
* Something went wrong during a ramp, I need to turn off the supply!
141 220 Flor de maria Blaszczyk
There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, *stop the ramp by clicking on the STOP button*. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and *don't forget to turn off the supply!*
142 219 Flor de maria Blaszczyk
143 221 Flor de maria Blaszczyk
*Even if the controls appear not to respond, set the voltage to 0 and turn the supply off*. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken.
144 219 Flor de maria Blaszczyk
145 219 Flor de maria Blaszczyk
* ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov  
146 219 Flor de maria Blaszczyk
* VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov
147 219 Flor de maria Blaszczyk
148 217 Flor de maria Blaszczyk
h2. Synoptic troubleshooting
149 217 Flor de maria Blaszczyk
150 217 Flor de maria Blaszczyk
%{color:red}"Synoptic Troubleshooting":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#Troubleshooting%
151 217 Flor de maria Blaszczyk
152 217 Flor de maria Blaszczyk
153 68 Michael Kordosky
h1. Hardware Connections and Configurations
154 1 William Badgett
155 141 William Badgett
* [[TPC Channel Mapping]]
156 51 Jason St. John
* [[Non-TPC Detectors]] *Naming and numbering in MC7*
157 133 Jason St. John
* "Cable Patch Panel Spreadsheet":http://goo.gl/JXiH8Y
158 54 Jason St. John
* [[Connections on the Control Room Patch Panel]]
159 54 Jason St. John
160 51 Jason St. John
* [[Logical inputs to the CAEN V1740]]
161 51 Jason St. John
* [[Connections into the WUT TDC]]
162 51 Jason St. John
* [[Connections into the V1751 Digitizer]]
163 1 William Badgett
* [[Trigger inputs|Connections in/out of the V1495 Trigger]]
164 68 Michael Kordosky
* [[Logic pulse widths and discriminator thresholds]]
165 68 Michael Kordosky
* [[WUT configuration]]
166 74 William Badgett
* [[Timing of ACnet TCLK Events]]
167 140 William Badgett
* The [[Teensy]] LAr ASIC controller's configuration
168 50 Michael Kordosky
169 138 Roberto Acciarri
* [[Punch Through counters voltages and rates ]]
170 138 Roberto Acciarri
171 102 Jason St. John
h1. Electronic Hardware Documentation
172 45 Michael Kordosky
173 51 Jason St. John
* Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat
174 51 Jason St. John
* [[LariatOnlineGit]] Accessing the LArIAT Online GIT repository
175 98 Brian Rebel
* [[Building LArIATOnline with CMake]]
176 44 Michael Kordosky
177 51 Jason St. John
* [[LArIAT TPC Readout Parameters]]
178 185 William Badgett
* [[LArASIC Preamp Documentation from BNL]]
179 51 Jason St. John
* [[LArIAT DAQ Commercial Products]] Technical Documentation and Firmware Files
180 101 William Badgett
* [[LArIAT CAEN Products]] Technical Documentation and Firmware Files
181 51 Jason St. John
* Test Beam Wire Chamber Sten Hensen's [[TDC Readout Documentation]]
182 89 William Badgett
* WUT Wave Union TDC: [[Jinyuan's TDC Documentation]]
183 51 Jason St. John
* [[v1495 notes]] Notes and working area for the V1495 and V2718
184 195 Johnny Ho
* [[A3818 PCI optical controller]] links
185 35 Michael Kordosky
186 1 William Badgett
* [[DAQ Configuration]] settings, parameters, starting a run
187 51 Jason St. John
* [[Raw Data Format and Readout Source code]]
188 100 John Freeman
* [[All things lariat-artdaq]] (UPDATED 11/5/14)
189 154 Jason St. John
* [[Interpreting Wire Chamber TDC Data]]
190 36 Michael Kordosky
191 54 Jason St. John
* [[LArIAT Test Stand]] Photographs
192 47 Michael Kordosky
193 72 Michael Kordosky
h1. Diagnostic and Control Software
194 72 Michael Kordosky
195 73 Michael Kordosky
_Test programs in the_ lariat-online/daq/src _directory_
196 73 Michael Kordosky
197 72 Michael Kordosky
* [[DumpDigitizerData]]
198 72 Michael Kordosky
* [[SummarizeDAQOutput]]
199 72 Michael Kordosky
* [[Reset1495]]
200 72 Michael Kordosky
* [[ScalarReadout]]
201 97 Pawel Kryczynski
* [[Data_quality_monitoring_module]]
202 72 Michael Kordosky
203 139 William Badgett
* CAEN's [[WaveDump]] utility
204 139 William Badgett
205 75 Jason St. John
h1. Examining the WC mini DAQ ASCII data
206 75 Jason St. John
207 75 Jason St. John
The Processor makes TTrees from the ASCII files saved by @mc7-daq.py@
208 82 Jason St. John
[[WC MiniDAQ Data Processor]]
209 75 Jason St. John
210 153 Jason St. John
h1. Data Files from the year 2013
211 76 Will Flanagan
212 85 William Badgett
Data files for various beam conditions can be found in the [[DataFiles]] page.
213 85 William Badgett
214 51 Jason St. John
h1. Meetings
215 51 Jason St. John
216 51 Jason St. John
* [[Connecting to LArIAT ReadyTalk]]
217 198 Johnny Ho
218 1 William Badgett
h1. Miscellaneous
219 198 Johnny Ho
220 216 Luke Batten
* The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov
221 198 Johnny Ho
* [[Proxy using SSH tunnel]]