LArIAT Run 2 Shifter Instructions » History » Version 69

Version 68 (Jason St. John, 04/07/2016 10:33 AM) → Version 69/119 (Jason St. John, 04/07/2016 10:52 AM)

h1. LArIAT Run 2 Shifter Instructions

*_Remember to call MCR (x3721) at the beginning and ending of each shift!!!_*
Main Control Room (internationally) +1.630.840.3721

h2. Please don't hesitate to contact the experts when something doesn't look right!!!
%{color:red}"Troubleshooting Problems":

[[Before your first shift at LArIAT]] check here so you're ready.

h1. Log onto the shift computer

h2. For a shift at Fermilab:

* From an MCenter machine, such as ftbflx18 or ftbflx15:
** Log in with username = @ftbf_user@.
* From a ROC West machine, such as lariat-cr-02:
** Log in with username = @lariat@.
* Password is YOUR kerberos password, then it will ask for YOUR kerberos principal. (Reverse from usual order)

h2. From a remote shift:

* Start the Fermilab Virtual Private Network (available "here": )
** username and password: your own Services Account.
* Get a kerberos ticket on your machine with @kinit@
* Open an ssh tunnel to the shift computer<pre>ssh -C -L 5900:localhost:5900 </pre>
* Direct your VNC viewer to localhost:5900. Password is the same as the docdb password. We recommend RealVNC but others are worth trying in case they work better for you.

h1. Refresh the kerberos ticket on the shift computer (NOT the DAQ machine!)

* Once logged in, open a terminal window. If "klist" doesn't show that you have a valid kerberos ticket, get one: @kinit <your_kerberos_principal>@
* If ftbf_user (MCenter) or lariat (ROC West) is already logged in - reset the kerberos ticket: kdestroy and then @kinit <your_kerberos_principal>@
* Be careful you're that you're on the correct terminal and _not_ on a DAQ terminal when you issue kdestroy.

h1. Getting your setup ready

* Check the pressure in the wire chambers’ gas bottle: bottle and the slow bubbles in the wire chambers’ gas panel “Return” bubblers. If you don’t know how to do it or where it is, shame on us we haven’t showed you yet (see [[Wire Chamber Gas/Electronics Instructions]] Instructions]]).
** You need this for the first e-log entry, the start of shift form.
If the pressure is less than 18 psi, please email the Run Co-ordinator at

* +ACNET/Synoptic+ does not always keep running properly after such reset, so it's recommended to restart +both of them+ (see instructions in the next point)

h2. Web pages to keep visible

* "Run Status":
* "Online DQM Data Stream Metrics": to see data block flow rate over the last 16 hours for all front-ends
* "Online DQM Data Stream": to see the data block counts and timing for the current run

h3. More useful web pages

* "Run Summary": of recent runs
* "Using ACNET for monitoring": This page contains the instructions to monitor the beam status through ACNET.
* "Synoptic instructions":
* "MCR Operations Elog": +in the new tab+ This is the Main Control Room (MCR) Elog. Use it to check any possible problem affecting the MCenter beam.
* "LArIAT Elog": Log the previous shifter out (if they are logged in), and log yourself in. Make a new entry with the "Start of Shift Checklist" form.

h2. %{color:green}During the shift%

* If you have questions, call an expert! "LArIAT Call List":
* Or, you can use Google Hangouts: send an email to to get invitation.
* As well as Google Hangouts, you can Skype an expert! How to access Skype:
* On the ftbf_user or lariat account:
** "cd ~/Downloads/skype-"
** "skype"
* You can find the password in the top right blue box of the whiteboard in the control room.

What beam should I request? The "Run Plan": lists current and upcoming beam requests, along with the other MCenter activities going on.

h3. Starting a new run

* Open in a new tab [[LArIAT DAQ Instructions]] and follow the instructions to start a new run.
* The DAQ should make an e-log entry for your automatically.
* Special cases:
** *Change/check the DAQ configuration* see "DAQ Configuration Instructions":
** *Changing gain or filter times* in the ASICS: check that there was [[Teensy Output]] in the initialization spew from the DAQ.

h3. During an ongoing run

* Monitor the beam status using the instructions in the "Using ACNET for monitoring":, namely
** the *S60* page of ACNET and
** the profile plots for *MC7WC1* and *MC7WC2*.
** From ~ftbf_user/acnet or ~lariat/acnet, run @./ to open the Synoptic displays.

* "Run Status Webpage": to monitor the run status (run number, number of spills, number of triggers, etc)
* Note that, on the Run Status Webpage, %{color:red}if the run time appears red%, and the spills stop increasing, the run has come to a stop. It will also speak an audible alert, once.
** In order to get it running again, you can issue the 'lariatReset' command in the DAQ terminal, followed by 'go'.
** Sometimes the Run Status page freezes. If this happens, see if your DAQ command line has reset. If so, the run has ended. Follow above instructions.
* Did the run number fail to increment? If the number has reset to a low number such as 1 stop the run and then edit /opt/lariat-online/config/runNumber.dat to correct the run number. Start new run after.
* "Online DQM": to check basic data integrity.
** If the rate of MWPCs data blocks during the beam spill is not very close to the data block rates of the other systems, debugging this is urgent. !>GoodMWPCsSmall.gif!
*** On the Data Blocks page, monitor the *"Data Block Time Stamps"* graph. During the beam (1-5 s) the MWPC line should follow a similar line shape to the V1740 and V1751 lines, and not be too far below.
*** You can isolate plots by hovering over the legend.
* Use the event viewer to see TPC events. See "Event Viewer Instructions": . If you +decide to post some interesting event+ in the e-log, +please write the number of the spill/event in the title or content of the entry+. It's much easier to search for them later!
* Useful: "Recent run history": , python scripts for making [[DQM plots]]

h1. Troubleshooting

Most problems solved: "Troubleshooting Page": doesn't help, "call an expert":

* If a VME Power Supply has to be turned off in an emergency situation (e.g. problem with TPC PMTs) - turn off the regulate function first, then put the decreasing voltage values in the quick ramp window, go to 0 and then turn off!

h2. %{color:green}At the end of the shift%

* Make an entry in the e-Log using the form "Shift Summary" summing up the relevant information from the shift (make sure to include the total trigger counts for the V1751s and V1740s). For any other information to be posted in the e-Log during the shift use the "default" form.
# Call MCR to stop the beam

h2. *_Remember to call MCR at the beginning and ending of each shift!!!_*

h1. *_Remember to call MCR at the beginning and ending of each shift!!!_*