Wiki » History » Version 324
« Previous -
Version 324/332
(diff) -
Next » -
Current version
Ron Rechenmacher, 11/16/2017 03:03 PM
LArIAT shift: +1 9293 LARIAT (= +1 929 352 7428)
ROC West x5413
MCenter Control Room (MC-CR) x3726 or x8871
Internationally: prefix +1 630 840 to the above
Shifter Stuff¶
- Before your first shift at LArIAT
- If remote or starting local shift from scratch: How to Set up your Shift Station
- LArIAT Run 3 Shifter Instructions (April 2016)
- Run Plan
- LArIAT Expert Call List (User is "lariat", password same as docdb)
- Troubleshooting Page
- Table of contents
- Shifter Stuff
- Troubleshooting
- VNC Shared Desktop
- Trigger and DAQ system
- Common failures:
- Wire Chambers
- High voltage for the wire chambers, TPC wireplane bias, cryo PMTs
- SiPM and ARAPUCA power supply
- Event Viewer
- Power Cycle the ASICs (TPC Wire Noise)
- Power Cycle the pLAPPD V1742 (V1751 or V1742 readout hang ups)
- Recover from a LAr level interlock
- Synoptic troubleshooting
- DQM troubleshooting
- PB S60 PPD Params
- Hardware Connections and Configurations
- Electronic Hardware Documentation
- Diagnostic and Control Software
- Examining the WC mini DAQ ASCII data
- Data Files from the year 2013
- Meetings
- Miscellaneous
- Commissioning tests before Run III
- November 2017 too many spills per subrun issue
Link summary ( quick start )¶
- Live event viewer snapshots
- Accelerator Schedule (power outages, etc)
- LArIAT eLog For making/ending access, starting/ending shifts.
- LArIAT DAQ Instructions (February 2016)
- LArIAT Run Status Webpage
- LArIAT Run Summary Webpage
- LArIAT Online DQM (works with VPN or VNC or on-site only, for now)
- How to use the event viewers
- Synoptic
- Beam monitoring
- Beams Division E-Log
- MC7 WebCam
- How to take a Teensy Channel Scan run
Important knowledge¶
- List of MC7 ACNET devices
- MC-7 Enclosure Layout
- In case the beam TVs are off
- The LArIAT supercycle How DAQ operations relate to the main injector supercycle
- Hardware-Connections-and-Configurations
- Electronic Hardware Documentation
Obsolete links¶
- LArIAT Run 1 Shifter Instructions (June 2015)
LArIAT Engineering Run (Aug 2014) How to Run the Lariat Data Acquisition(OBSOLETE)Shifter Instructions August 2014(Some obsolete information)How to make Lariat DAQ Plots(OBSOLETE)
Troubleshooting¶
Any time day or night, please call experts if you are in need of help and the answer is not provided below.
VNC Shared Desktop¶
Mouse clicks not working. Keyboard remapped. Arrow keys type numbers.¶
Try this first
Press the modifier keys one by one (Alt/Option, "Windows", Control or "Apple"/command, Alt/Option, Control) and see if this solves the problem. This has been suggested as a solution by various online forums, for when the keyboard is acting like a modifier key is stuck in VNC.
Try this second
Kill synoptic, restart it.
Procedure:- Open a new terminal window on
lariat-cr-02
(or log in remotely). - Find the process ID's to kill by issuing the command
ps aux | grep javaws
- The integer after
lariat
is the process ID. For example:[lariat@lariat-cr-02 ~]$ ps aux | grep javaws lariat 1319 19.3 12.2 4049912 471044 pts/1 Sl Jul15 777:36 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=38857 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawO3BJLo lariat 9599 4.0 12.5 4048312 482184 pts/1 Sl Jul14 243:28 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=52805 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javawBDjN44 lariat 13226 0.0 0.0 103312 868 pts/3 S+ 10:53 0:00 grep javaws lariat 22629 9.9 7.7 4044088 298244 pts/1 Sl Jul17 91:52 /home/lariat/jdk1.8.0_91/jre/bin/java -classpath /home/lariat/jdk1.8.0_91/jre/lib/deploy.jar -Djava.security.policy=file:/home/lariat/jdk1.8.0_91/jre/lib/security/javaws.policy -DtrustProxy=true -Xverify:remote -Djnlpx.home=/home/lariat/jdk1.8.0_91/jre/bin -Djava.security.manager -Djnlpx.origFilenameArg=/home/lariat/acnet/LArIAT_mainmenu -Djnlpx.remove=true -Dsun.awt.warmup=true -Xbootclasspath/a:/home/lariat/jdk1.8.0_91/jre/lib/javaws.jar:/home/lariat/jdk1.8.0_91/jre/lib/deploy.jar:/home/lariat/jdk1.8.0_91/jre/lib/plugin.jar -Xincgc -Djavaws.gov.fnal.controls.kerberos.enable=true -Dsun.java2d.noddraw=true -Xms32m -Xmx256m -Djnlpx.heapsize=32m,256m -Djnlpx.splashport=35726 -Djnlpx.jvm=/home/lariat/jdk1.8.0_91/jre/bin/java -Djnlpx.vmargs=LVhpbmNnYwAtRGphdmF3cy5nb3YuZm5hbC5jb250cm9scy5rZXJiZXJvcy5lbmFibGU9dHJ1ZQAtRHN1bi5qYXZhMmQubm9kZHJhdz10cnVlAA== com.sun.javaws.Main /tmp/javaw8bLkDI
- Kill all of these processes: (you may ignore the one for your
grep
)kill -9 1319 kill -9 9599 kill -9 22629
- Check that your keyboard and mouse function are normal again.
- Restart DAQ or Event Viewer if needed. THEN re-launch Synoptic.
Try this third
Ask on the hang-out if someone, via VNC, is able to use the keyboard ... sometimes it works.
Try this fourth
Kill VNC. Restart it.
'VNC server closed connection' might mean that the vnc server on lariat-cr-02 is not running.
Log on to lariat-cr-02 and check if the X11 vnc sever is running:
$ ps aux | grep vnc lariat 31537 0.0 0.0 103308 860 pts/5 S+ 00:12 0:00 grep vnc
Re-launch it:
[lariat@lariat-cr-02 ~]$ x11vnc -create -localhost -xinerama -display :0 -shared -alwaysshared -many -forever -bg -rfbauth ~/.x11vnc/passwd
Check that it is running now:
[lariat@lariat-cr-02 ~]$ ps aux | grep vnc lariat 31670 8.4 1.6 194032 61944 ? Ss 00:14 0:10 x11vnc -create -localhost -xinerama -display :0 -shared -many -forever -bg -rfbauth /home/lariat/.x11vnc/passwd lariat 31753 0.0 0.0 103308 860 pts/5 S+ 00:16 0:00 grep vnc
...and shifter is able to connect (the most important part!)
If nothing else works
Someone has to go to ROC-w and hard reboot the PC...
Trigger and DAQ system¶
When the DAQ crashes, please post the relevant information to DAQerrors for experts to study.
Common failures:¶
Communication failure with the front ends.- Trying clearing it with
lariatReset
or./lariat-online/daq/bin/lariatReset
Wait a few minutes and then try thego 500
command again.
lost connection Permission denied (gssapi-keyex,gssapi-with-mic)
- Log out of the DAQ machine, and use the command
kticket
to pick up a fresh kerberos ticket with the special principal that we use for the DAQ:lariat/lariat@lariat-daq00.fnal.gov@FNAL.GOV
. Check for this success withklist
. Then log back in to the DAQ machine and start a new run.
ERROR: ReadRegisgter ACQUISITION_STATUS CAEN_DGTZ_CommError BoardId 0
- Wait 2 minutes, then try to start again.
- Try issuing the command
lariatReset 0 1
waiting more then 5 seconds, and trying again.
This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error.
Found persistent communication port still open
- Wait 8 minutes for the ports to clear, then try again.
An exception occurred when trying to send a message to ..."
- Check that the terminal you are running the DAQ from is indeed lariat-daq00 (or a screen within it). Output from
klist
should show the default principal as "lariat/lariat/lariat-cr-02.fnal.gov@FNAL.GOV" or similar if running from a screen off lariat-daq00 from the ROC-West station.
- Stop the run and then on the DAQ terminal edit
/opt/lariat-online/config/runNumber.dat
to correct the run number. Start new run after.
Run Status page not updating¶
- Try refreshing the page.
- If the ACNET node has gone down (as in a power loss) may need to re-start the tomcat servlet:
ssh tomcat@lariat-wbm.fnal.gov
, thenkitty
. (Yep, "kitty.")
- If there are messages of lost connection and failed to copy run.xml.new :
- kill the run (by hitting enter in the DAQ terminal), type the command "kticket" into the DAQ terminal, the start a new run by typing the command "go 500".
LArIAT beam mode not automatically updating Beam / No Beam States¶
- Try restarting the script on an ACnet console, page D129
- Select the Restart Persistent option
- From list of processes, select LArIAT_beamCheck.acl
- Confirm by clicking the Restart button
Other failures:¶
- If the DAQ is having trouble talking to the crates, try issuing the command
lariatReset 0 1
waiting more then 5 seconds, and trying again.
This command is appropriate when one sees any type of "CAEN_DGTZ" error. This command will NOT help the persistent ports error. - No triggers at all If the trigger crate has been power cycled and the V1495 is not emitting triggers: Reset1495
Wire Chambers¶
Nominal Voltages:WC1 | -2425 |
---|---|
WC2 | -2400 |
WC3 | -2425 |
WC4 | -2450 |
- If the wire chambers HV trips: Wire Chamber voltage instructions to control wire chamber voltages
- If the wire chambers are acting strangely: Wire Chamber Electronics remote power cycling instructions including mini-DAQ and telnet interface.
- If the wire chamber system is skipping spill numbers, it may be a high trigger rate (> 30 k per spill). The wire chamber controller may still be reading out the hits from the last spill.
High voltage for the wire chambers, TPC wireplane bias, cryo PMTs¶
ICathode HV trips off due to LAr level interlock.
Drift HV TPC Cathode Voltage
General Instructions for controls via Synoptic
Also valid for some of the muon range stack paddles (1-8), the halo veto and 1 aerogel PMT.
ONLY IF SYNOPTIC DOESN'T WORK Instructions for controls via ACNET
- The readback voltage doesn't make any sense, is of the wrong polarity and/or is way above the nominal value, but just for a fraction of time.
This is most likely just a glitch in the readout, please ignore if it happens once or twice. If it happens too frequently, set voltage to 0, turn off the power supply and report it.
- The readback voltage doesn't make any sense, is of the wrong polarity and is way above the nominal value, and the value is stable~ish (you might see small fluctuations over time)
This is a typical symptom of a bad power supply. Try to set it back to 0, turn it off and disconnect the SHV cable. If the readout values remain the same, it confirms there is a hardware problem. Please contact Brian Fellenz for repairs if we don't have a spare for that unit.
FYI, the power supplies have a given voltage rating and cannot physically deliver voltages beyond that range. Also, their polarities cannot be swapped so if you see the wrong sign on the readout ie a positive voltage on a negative power supply or vice versa, then you can be sure the unit is broken or you have a problem somewhere else in the system.
- Bias voltage rating: - 500 V for the shield and induction planes, + 500 V for the collection plane.
- ELT and Hamamatsu PMTs: +2000 V
- Wire chambers: - 5000 V
- All of the other units: -2000 V
- Something went wrong during a ramp, I need to turn off the supply!
There is an interlock that prevents you from using the normal fast controls on the power supply when a ramp is on-going. First, stop the ramp by clicking on the STOP button. This will stop the ramp and clear the interlock. If you need to go back to 0 quickly, use the fast controls to set the voltage to 0 and don't forget to turn off the supply!
Even if the controls appear not to respond, set the voltage to 0 and turn the supply off. Contact Charlie Briegel first and describe the problem in detail. If you can and if it's safe, disconnect the SHV cable from the power supply. If the readback values remain the same, then the power supply is broken.
- ACNET controls expert: Charles (Charlie) Briegel, x4510 briegel@fnal.gov
- VME power supplies repairs / replacements: Brian Fellenz, x2512 fellenz@fnal.gov
SiPM and ARAPUCA power supply¶
A low-voltage power supply module on the MCenter control room racks is used to power SiPM boards and (as of Run III) the ARAPUCA light detection device.
To turn it on:- Switch on the supply
- Press "Output on/off" to turn on output
- Press "Voltage"
- Adjust to 24V using either the knob (left) or the number pad (right)
Event Viewer¶
- If the event viewer stops cycling through the events and the terminal from which it is running shows a list of lines saying something like "event XX in file, but on event 0" with XX any number,
- ctrl+z
- ps -def to look at the processes list. Look for the job ID of the process "python /lariat/app/users/EventViewer/..."
- kill -9 JobID to kill the event viewer
- open a new event viewer
ssh lariat@lariatgpvm01.fnal.gov cd /lariat/app/users/EventViewer/; source SETUP_ONLINE_EVENTVIEWER.sh
- If the problem persists and/or become more frequent, exit the session you are in and try to connect from a different gpvm machine
Power Cycle the ASICs (TPC Wire Noise)¶
- Anecdotal evidence has shown that analog noise on the TPC wire readout channels can be
alleviated by power cycling the TPC electronics: ASICs and Warm Receiver/Drivers - You can remotely power cycle the TPC electronics. Log onto lariat-gateway00.fnal.gov with X-forwarding enabled, type "firefox", and go to http://lariat-pdu02.fnal.gov. Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7). Choose the CYCLE option, which will automatically give a 60 second powerless interlude.
- Further important informations can be found on the white board in MC-CR.
- This mechanism is controlled by a smart power distribution unit (PDU) located under the
TPC electronics power supply near the cryostat in the enclosure. Due to electrical safety
rules you must NOT use any of the other outlets on this PDU. Do NOT remove the protective
outlet covers at any time.
Power Cycle the pLAPPD V1742 (V1751 or V1742 readout hang ups)¶
Error looks like this:
*** Fatal error opening CAEN board at 3,0
- Most likely the V1742 has hung up and needs a power cycle.
- You can remotely power cycle the V1742 digitizers. On a firefox on lariat-cr-01 or at MCenter, go to http://ftbf-pdu05.fnal.gov. Usename "ftbf" (password posted on Post-It on side of ROC monitor, and whiteboard in MC7, or docdb 2187). Choose the CYCLE option, which will automatically give a 30 second powerless interlude.
- This mechanism is controlled by a smart power distribution unit (PDU) located under pLAPPD readout
crate near the beam line on the upstream side.
Recover from a LAr level interlock¶
Assuming the LAr Level dropped low enough to trip off the TPC Cathode power supply, and maybe you even powered down the PMTs.
- Be sure HV is set to zero for these: Cathode Drift, Wire Bias, cryo-PMTs
How to control the Drift HV - Power down the ASICs. (They should be off when TPC voltages change.)
- Contact one of these people to Reset the HV LAr level interlock in IFIX
- Bring up the Cathode
- Bring up the Wire Bias
- Bring up the ASICs
- Bring up the PMTs
Synoptic troubleshooting¶
DQM troubleshooting¶
If the DQM is flashing red...
- Check to see if the DAQ is running. If it isn't running, you should start a new run!
- Check the LArIAT Run Status page. If the number under Complete Files Waiting is greater than 10, please call a data handling expert!
- Check the disk space on the
/lariat/data
mount with the commanddf -h /lariat/data
on alariatgpvm
orlariat-daq
machine. If the disk space is 100% full, please call a DQM expert!
Some details of how DQM is set up
PB S60 PPD Params¶
If F:MC7AN , F:MC7ANB and F:MC7ANP show the values 1 -6 (in red), this is probably a communication problem, where we are not receiving data from them.
Usually it comes back in a few minutes.
If not, call the MCR and ask them if they can see those values.
Hardware Connections and Configurations¶
- TPC Channel Mapping
- Non-TPC Detectors Naming and numbering in MC7
- Cable Patch Panel Spreadsheet as well as high voltage settings, power supplies, and warm PMT operating curves
- Logical inputs to the CAEN V1740
- Connections into the WUT TDC
- Connections into the V1751 Digitizer
- Connections in/out of the V1495 Trigger
- Logic pulse widths and discriminator thresholds
- WUT configuration
- Timing of ACnet TCLK Events
- The Teensy LAr ASIC controller's configuration
Electronic Hardware Documentation¶
- Link to LArIAT eLog http://dbweb0.fnal.gov/ECL/lariat
- LariatOnlineGit Accessing the LArIAT Online GIT repository
- Building LArIATOnline with CMake
- LArIAT TPC Readout Parameters
- LArASIC Preamp Documentation from BNL
- LArIAT DAQ Commercial Products Technical Documentation and Firmware Files
- LArIAT CAEN Products Technical Documentation and Firmware Files
- Test Beam Wire Chamber Sten Hensen's TDC Readout Documentation
- WUT Wave Union TDC: Jinyuan's TDC Documentation
- v1495 notes Notes and working area for the V1495 and V2718
- V1495 docs on DocDB, built from source:fpga/doc/
- A3818 PCI optical controller links
- DAQ Configuration settings, parameters, starting a run
- Raw Data Format and Readout Source code
- All things lariat-artdaq (UPDATED 11/5/14)
- Interpreting Wire Chamber TDC Data
- LArIAT Test Stand Photographs
Diagnostic and Control Software¶
Test programs in the lariat-online/daq/src directory
- CAEN's WaveDump utility
Examining the WC mini DAQ ASCII data¶
The Processor makes TTrees from the ASCII files saved by mc7-daq.py
WC MiniDAQ Data Processor
Data Files from the year 2013¶
Data files for various beam conditions can be found in the DataFiles page.
Meetings¶
Miscellaneous¶
- The control room shifter computer (MCenter) can be logged into as ftbf_user@ftbflx18.fnal.gov after passing through lariat-gateway.fnal.gov
- Proxy using SSH tunnel
Commissioning tests before Run III¶
- TOF calibration with electrons - Robb Carey
If others want to do calibration, please tell me.