Project

General

Profile

History

Wiki » History » Version 275

« Previous - Version 275/332 (diff) - Next » - Current version
Jason St. John, 07/07/2016 12:40 AM

LArIAT shift: +1 9293 LARIAT (= +1 929 352 7428)
ROC West x5413
MCenter Control Room (MC-CR) x3726 or x8871
Internationally: prefix +1 630 840 to the above

Shifter Stuff

Link summary ( quick start )

Important knowledge

Obsolete links

Troubleshooting

Any time day or night, please call experts if you are in need of help and the answer is not provided below.

VNC Shared Desktop

'VNC server closed connection' might mean that the vnc server on lariat-cr-02 is not running.

Log on to lariat-cr-02 and check if the X11 vnc sever is running:

$ ps aux | grep vnc 
lariat 31537 0.0 0.0 103308 860 pts/5 S+ 00:12 0:00 grep vnc

Re-launched it:

[lariat@lariat-cr-02 ~]$ x11vnc -create -localhost -xinerama -display :0 -shared -many -forever -bg -rfbauth ~/.x11vnc/passwd

Checked that it is running now:

[lariat@lariat-cr-02 ~]$ ps aux | grep vnc
lariat 31670 8.4 1.6 194032 61944 ? Ss 00:14 0:10 x11vnc -create -localhost -xinerama -display :0 -shared -many -forever -bg -rfbauth /home/lariat/.x11vnc/passwd
lariat 31753 0.0 0.0 103308 860 pts/5 S+ 00:16 0:00 grep vnc

...and shifter is able to connect (the most important part!)

Trigger and DAQ system

When the DAQ crashes, please post the relevant information to DAQerrors for experts to study.

Common failures:

  • Communication failure with the front ends. Trying clearing it with 
    lariatReset

    Wait a few minutes and then try the go 480 command again.
  • Unable to start a new run: 
    lost connection
Permission denied (gssapi-keyex,gssapi-with-mic)
    • Log out of the DAQ machine, and use the command kticket to pick up a fresh kerberos ticket with the special principal that we use for the DAQ: lariat/lariat@lariat-daq00.fnal.gov@FNAL.GOV. Check for this success with klist. Then log back in to the DAQ machine and start a new run.
  • DAQ fails with ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0 wait 2 minutes, then try to start again.
    • Try issuing the command lariatReset 0 1 waiting more then 5 seconds, and trying again.
      This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error.
  • DAQ fails with Found persistent communication port still open
    • Wait 8 minutes for the ports to clear, then try again.
  • DAQ fails with "An exception occurred when trying to send a message to ..."
    • Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it). Output from klist should show the default principal as "" or similar if running from a screen off lariat-daq00 from the ROC-West station.
    • Run number failed increment or has reset to a low number such as 1: Stop the run and then on the DAQ terminal edit /opt/lariat-online/config/runNumber.dat to correct the run number. Start new run after.

Other failures:

  • If the DAQ is having trouble talking to the crates, try issuing the command lariatReset 0 1 waiting more then 5 seconds, and trying again.
    This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error.
  • No triggers at all If the trigger crate has been power cycled and the V1495 is not emitting triggers: Reset1495
  • Run Status page not updating
    • Try refreshing the page.
    • If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet: ssh tomcat@lariat-wbm.fnal.gov, then kitty. (Yep, "kitty.")

Wire Chambers

Nominal Voltages:
WC1 -2425
WC2 -2400
WC3 -2425
WC4 -2450

High voltage for the wire chambers, TPC wireplane bias, cryo PMTs

ICathode HV trips off due to LAr level interlock.
Drift HV TPC Cathode Voltage

General Instructions for controls via Synoptic

Also valid for some of the muon range stack paddles (1-8), the halo veto and 1 aerogel PMT.

ONLY IF SYNOPTIC DOESN'T WORK Instructions for controls via ACNET

  • The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time.
    This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it.
  • The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time)
    This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit.

FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system.
- Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane.
- ELT and Hamamatsu PMTs: +2000 V
- Wire chambers: - 5000 V
- All of the other units: -2000 V

  • Something went wrong during a ramp, I need to turn off the supply!
    There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, stop the ramp by clicking on the STOP button. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and don't forget to turn off the supply!

Even if the controls appear not to respond, set the voltage to 0 and turn the supply off. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken.

Event Viewer

  • If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like "event XX in file, but on event 0" with XX any number,
    • ctrl+z
    • ps -def to look at the processes list. Look for the job ID of the process "python /lariat/app/users/EventViewer/..."
    • kill -9 JobID to kill the event viewer
    • open a new event viewer 
      ssh lariat@lariatgpvm01.fnal.gov 
cd /lariat/app/users/EventViewer/; source SETUP_ONLINE_EVENTVIEWER.sh
  • If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine

Power Cycle the ASICs (TPC Wire Noise)

  • Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be
    alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers
  • You can remotely power cycle the TPC electronics. Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov. Usename "ftbf" (password posted on whiteboard in MC7). Choose the CYCLE option, which will automatically give a 60
    second powerless interlude.
  • Further important informations can be found on the white board in MC-CR.
  • This mechanism is controlled by a smart power distribution unit (PDU) located under the
    TPC electronics power supply near the cryostat in the enclosure. Due to electrical safety
    rules you must NOT use any of the other outlets on this PDU. Do NOT remove the protective
    outlet covers at any time.

Recover from a LAr level interlock

Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs.

  1. Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs
    How to control the Drift HV
  2. Power down the ASICs. (They should be off when TPC voltages change.)
  3. Bring up the Cathode
  4. Bring up the Wire Bias
  5. Bring up the ASICs
  6. Bring up the PMTs

Synoptic troubleshooting

Synoptic Troubleshooting

DQM troubleshooting

If the DQM is flashing red...

  • Check to see if the DAQ is running. If it isn't running, you should start a new run!
  • Check the LArIAT Run Status page. If the number under Complete Files Waiting is greater than 10, please call a data handling expert!
  • Check the disk space on the /lariat/data mount with the command df -h /lariat/data on a lariatgpvm or lariat-daq machine. If the disk space is 100% full, please call a DQM expert!

PB S60 PPD Params

If F:MC7AN , F:MC7ANB and F:MC7ANP show the values 1 -6 (in red), this is probably a communication problem, where we are not receiving data from them.
Usually it comes back in a few minutes.
If not, call the MCR and ask them if they can see those values.

Hardware Connections and Configurations

Electronic Hardware Documentation

Diagnostic and Control Software

Test programs in the lariat-online/daq/src directory

Examining the WC mini DAQ ASCII data

The Processor makes TTrees from the ASCII files saved by mc7-daq.py
WC MiniDAQ Data Processor

Data Files from the year 2013

Data files for various beam conditions can be found in the DataFiles page.

Meetings

Miscellaneous