Project

General

Profile

Wiki » History » Version 243

Version 242 (Jason St. John, 04/07/2016 05:30 PM) → Version 243/332 (Jason St. John, 04/08/2016 09:22 AM)

!https:/redmine/attachments/download/17373/Lariat_Logo_256.jpg!

ROC West x5413
MCenter Control Room (MC-CR) x3726 or x8871
Internationally: prefix *+1 630 840* to the above

* Call the Main Control Room (x3721) at the start and at the end of shift. This is required, and pleasant.
* If the beam goes away for more than a few spills, look for Accelerator Alarms in ACNET (a default window). If the problem persists a few minutes, and you see nothing about it in the MCR elog, (or you just really want to find out more), call the MCR. They're friendly, though they may be super-busy.

{{toc}}

h1. Shifter Stuff

* [[Before your first shift at LArIAT]]
* [[LArIAT Run 2 Shifter Instructions]] (February 2016)
** [[Run Plan]]
** "LArIAT Expert Call List":http://lariat.fnal.gov/internal/internal.html (User is "lariat", password same as docdb)

h2. Link summary ( [[quick start]] )

* "Accelerator Schedule":http://www-ad.fnal.gov/ops/schedule.html (power outages, etc)
* "LArIAT e-Log":http://dbweb0.fnal.gov:8080/ECL/lariat/E/index For making/ending access, starting/ending shifts.
* [[LArIAT DAQ Instructions]] (February 2016)
* "LArIAT Run Status Webpage":http://lariat-wbm.fnal.gov/lariat/run-alert.html
* "LArIAT Run Summary Webpage":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary
* "LArIAT Online DQM":http://lariat-dqm.fnal.gov (works with VPN or VNC or on-site only, for now)
* How to use the [[event viewers]]
* How to make [[DQM plots]] (may be outdated)
* "Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring
* "Beam monitoring(ACNET directly)":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki
* "Accelerator Operations Elog":https://www-bd.fnal.gov/Elog/?logNames=Operations
* "MC7 WebCam":http://131.225.176.26/

h3. Important knowledge

* "List of MC7 ACNET devices":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/MC7_in_ACNET
* [[MC-7 Enclosure Layout]]
* "In case the beam TVs are off":http://www-bd.fnal.gov/notifyservlet/www?project=&refresh=on&infolinks=top
* [[The LArIAT supercycle]] _How DAQ operations relate to the main injector supercycle_
* "Hardware-Connections-and-Configurations":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Hardware-Connections-and-Configurations
* "Electronic Hardware Documentation":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Wiki#Electronic-Hardware-Documentation

h3. Obsolete links

* [[LArIAT Run 1 Shifter Instructions]] (June 2015)
* -LArIAT Engineering Run (Aug 2014) [[How to Run the Lariat Data Acquisition]]- (OBSOLETE)
* -"Shifter Instructions":https://docs.google.com/document/d/1mlN_UIOCqwhe0cFiU7mRc3-lxDZhY-4TmIy23rOzILI/pub August 2014- (Some obsolete information)
* -How to make [[Lariat DAQ Plots]]- (OBSOLETE)

h1. Troubleshooting

Any time day or night, please call experts if you are in need of help and the answer is not provided below.

h2. *Trouble getting started*

* If you cannot ssh to FTBFLX machines: Get a kerberos ticket for yourself: @kinit <yourKerberosPrincipal>@, then ssh ftbf_user@ftbflx18 (or ssh ftbf_user@ftbflx15 or ssh ftbf_user@ftbf-cr-03).
* If your Kerberos username and password are good, but you can't log on to FTBFLX[NN] machines: Have someone who CAN log into those machines check that your kerberos principal appears in @~ftbf_user/.k5login@ in the LArIAT section (email the Run Co-ordinator at stjohn@fnal.gov).
* The same ssh errors may also apply to computers at ROC West. Troubleshoot using the above methods.

h2. *Trigger and DAQ system*

When the DAQ crashes, please post the relevant information to [[DAQerrors]] for experts to study.



h3. Common failures:

* Unable to start a new run: <pre>lost connection
Permission denied (gssapi-keyex,gssapi-with-mic)</pre>
** Execute the command @$HOME/bin/$USER-kinit.sh@ to change the default principal to the special principal that we use for the DAQ: @lariat/lariat@lariat-daq00.fnal.gov@FNAL.GOV@. Check for this success with @klist@.
*
DAQ fails with @ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0@ wait 2 minutes, then try to start again.
** Try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error.
* DAQ fails with @Found persistent communication port still open@
** Wait 8 minutes for the ports to clear, then try again.
* DAQ fails with "An exception occurred when trying to send a message to ..."
** Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it). Output from @klist@ should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station.
** Run number failed increment or has reset to a low number such as 1: Stop the run and then on the DAQ terminal edit @/opt/lariat-online/config/runNumber.dat@ to correct the run number. Start new run after.



h3. Other failures:

* If the DAQ is having trouble talking to the crates, try issuing the command @lariatReset 0 1@ waiting more then 5 seconds, and trying again.
This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error.
* *No triggers at all* If the trigger crate has been power cycled and the V1495 is not emitting triggers: [[Reset1495]]
* *Run Status page not updating*
** Try refreshing the page.
** If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: @ssh tomcat@lariat-wbm.fnal.gov@, then @kitty@. (Yep, "kitty.")

h2. *Wire Chambers*

Nominal Voltages:
|_. WC1 | -2425 |
|_. WC2 | -2400 |
|_. WC3 | -2425 |
|_. WC4 | -2450 |

* If the wire chambers HV trips: "instructions for controls via Synoptic":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#4-Synoptic-power-supplies-controls
* If the wire chambers are acting strangely: "remote power cycling instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Communicating_with_the_wire_chambers including mini-DAQ and telnet interface.
* If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill). The wire chamber controller may still be reading out the hits from the last spill.

h2. *High voltage for the wire chambers, TPC wireplane bias, cryo PMTs*

%{color:red}Important! Follow instructions in link below if cathode HV trips off due to LAr level interlock.%
[[Drift HV]] TPC Cathode Voltage

"*Instructions for controls via Synoptic*":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#section-10

Also valid for some of the muon range stack paddles (1-8), the halo veto and 1 aerogel PMT.

*ONLY IF SYNOPTIC DOESN'T WORK* "Instructions for controls via ACNET":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_to_power_up_the_power_supplies

* The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time.
This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it.

* The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time)
This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit.

FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system.
- Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane.
- ELT and Hamamatsu PMTs: +2000 V
- All of the other units: -2000 V

* Something went wrong during a ramp, I need to turn off the supply!
There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, *stop the ramp by clicking on the STOP button*. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and *don't forget to turn off the supply!*

*Even if the controls appear not to respond, set the voltage to 0 and turn the supply off*. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken.

* *ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov*
* *VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov*

h2. *Event Viewer*

* If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like _"event XX in file, but on event 0"_ with _XX_ any number,
** _*ctrl+z*_
** _*ps -def*_ to look at the processes list. Look for the job ID of the process _"python /lariat/app/users/EventViewer/..."_
** _*kill -9 JobID*_ to kill the event viewer
** open a new event viewer

* If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine

h2. Power Cycle the ASICs (TPC Wire Noise)

* Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be
alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers
* You can remotely power cycle the TPC electronics. Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov. Usename "ftbf" (password posted on whiteboard in MC7). Choose the *CYCLE* option, which will automatically give a 60
second powerless interlude.
* Further important informations can be found on the white board in MC-CR.
* This mechanism is controlled by a smart power distribution unit (PDU) located under the
TPC electronics power supply near the cryostat in the enclosure. Due to electrical safety
rules you must *NOT* use any of the other outlets on this PDU. Do *NOT* remove the protective
outlet covers at any time.
!https://redmine.fnal.gov/redmine/attachments/download/25467/lariat-pdu02-cycle.png!

h2. Recover from a LAr level interlock

Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs.

# Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs
How to control the [[Drift HV]]
# Power down the ASICs. (They should be off when TPC voltages change.)
# Bring up the Cathode
# Bring up the Wire Bias
# Bring up the ASICs
# Bring up the PMTs

h2. Synoptic troubleshooting

%{color:red}"Synoptic Troubleshooting":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring#Troubleshooting%

h1. Hardware Connections and Configurations

* [[TPC Channel Mapping]]
* [[Non-TPC Detectors]] *Naming and numbering in MC7*
* "Cable Patch Panel Spreadsheet":http://goo.gl/JXiH8Y
* [[Connections on the Control Room Patch Panel]]

* [[Logical inputs to the CAEN V1740]]
* [[Connections into the WUT TDC]]
* [[Connections into the V1751 Digitizer]]
* [[Trigger inputs|Connections in/out of the V1495 Trigger]]
* [[Logic pulse widths and discriminator thresholds]]
* [[WUT configuration]]
* [[Timing of ACnet TCLK Events]]
* The [[Teensy]] LAr ASIC controller's configuration

* [[Punch Through counters voltages and rates ]]

h1. Electronic Hardware Documentation

* Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat
* [[LariatOnlineGit]] Accessing the LArIAT Online GIT repository
* [[Building LArIATOnline with CMake]]

* [[LArIAT TPC Readout Parameters]]
* [[LArASIC Preamp Documentation from BNL]]
* [[LArIAT DAQ Commercial Products]] Technical Documentation and Firmware Files
* [[LArIAT CAEN Products]] Technical Documentation and Firmware Files
* Test Beam Wire Chamber Sten Hensen's [[TDC Readout Documentation]]
* WUT Wave Union TDC: [[Jinyuan's TDC Documentation]]
* [[v1495 notes]] Notes and working area for the V1495 and V2718
* [[A3818 PCI optical controller]] links

* [[DAQ Configuration]] settings, parameters, starting a run
* [[Raw Data Format and Readout Source code]]
* [[All things lariat-artdaq]] (UPDATED 11/5/14)
* [[Interpreting Wire Chamber TDC Data]]

* [[LArIAT Test Stand]] Photographs

h1. Diagnostic and Control Software

_Test programs in the_ lariat-online/daq/src _directory_

* [[DumpDigitizerData]]
* [[SummarizeDAQOutput]]
* [[Reset1495]]
* [[ScalarReadout]]
* [[Data_quality_monitoring_module]]

* CAEN's [[WaveDump]] utility

h1. Examining the WC mini DAQ ASCII data

The Processor makes TTrees from the ASCII files saved by @mc7-daq.py@
[[WC MiniDAQ Data Processor]]

h1. Data Files from the year 2013

Data files for various beam conditions can be found in the [[DataFiles]] page.

h1. Meetings

* [[Connecting to LArIAT ReadyTalk]]

h1. Miscellaneous

* The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov
* [[Proxy using SSH tunnel]]