Project

General

Profile

LArIAT Run 2 Shifter Instructions » History » Version 69

Jason St. John, 04/07/2016 10:52 AM

1 2 Jason St. John
h1. LArIAT Run 2 Shifter Instructions
2 1 Jason St. John
3 40 Jason St. John
*_Remember to call MCR (x3721) at the beginning and ending of each shift!!!_*
4 54 Jason St. John
Main Control Room (internationally) +1.630.840.3721
5 1 Jason St. John
6 1 Jason St. John
h2. Please don't hesitate to contact the experts when something doesn't look right!!!
7 67 Jason St. John
%{color:red}"Troubleshooting Problems":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki#Troubleshooting%
8 1 Jason St. John
9 45 Jason St. John
[[Before your first shift at LArIAT]] check here so you're ready.
10 44 Jason St. John
11 50 Jason St. John
h1. Log onto the shift computer
12 1 Jason St. John
13 51 William Badgett
h2. For a shift at Fermilab:
14 48 Jason St. John
15 27 Luke Batten
* From an MCenter machine, such as ftbflx18 or ftbflx15:
16 1 Jason St. John
** Log in with username = @ftbf_user@.
17 27 Luke Batten
* From a ROC West machine, such as lariat-cr-02:
18 41 Jason St. John
** Log in with username = @lariat@.
19 48 Jason St. John
* Password is YOUR kerberos password, then it will ask for YOUR kerberos principal. (Reverse from usual order)
20 1 Jason St. John
21 50 Jason St. John
h2. From a remote shift:
22 48 Jason St. John
23 68 Jason St. John
* Start the Fermilab Virtual Private Network (available "here":https://vpn.fnal.gov/  https://vpn.fnal.gov )
24 68 Jason St. John
** username and password: your own Services Account. 
25 68 Jason St. John
* Get a kerberos ticket on your machine with @kinit@
26 68 Jason St. John
* Open an ssh tunnel to the shift computer<pre>ssh -C -L 5900:localhost:5900 lariat@lariat-cr-02.fnal.gov </pre>
27 68 Jason St. John
* Direct your VNC viewer to localhost:5900. Password is the same as the docdb password. We recommend RealVNC but others are worth trying in case they work better for you.
28 1 Jason St. John
29 50 Jason St. John
h1. Refresh the kerberos ticket on the shift computer (NOT the DAQ machine!)
30 1 Jason St. John
31 48 Jason St. John
* Once logged in, open a terminal window. If "klist" doesn't show that you have a valid kerberos ticket, get one: @kinit <your_kerberos_principal>@
32 48 Jason St. John
* If ftbf_user (MCenter) or lariat (ROC West) is already logged in - reset the kerberos ticket: kdestroy and then @kinit <your_kerberos_principal>@
33 1 Jason St. John
* Be careful you're that you're on the correct terminal and _not_ on a DAQ terminal when you issue kdestroy.
34 1 Jason St. John
35 50 Jason St. John
h1. Getting your setup ready
36 27 Luke Batten
37 69 Jason St. John
* Check the pressure in the wire chambers’ gas bottle: [[Wire Chamber Gas/Electronics Instructions]]
38 69 Jason St. John
** You need this for the first e-log entry, the start of shift form.
39 50 Jason St. John
** If the pressure is less than 18 psi, please email the Run Co-ordinator at stjohn@fnal.gov.
40 69 Jason St. John
41 1 Jason St. John
42 53 Jason St. John
h2. Web pages to keep visible
43 1 Jason St. John
44 66 Jason St. John
* "Run Status":http://lariat-wbm.fnal.gov/lariat/run.html
45 53 Jason St. John
* "Online DQM Data Stream Metrics":http://lariat-dqm.fnal.gov/metrics/data-stream to see data block flow rate over the last 16 hours for all front-ends 
46 53 Jason St. John
* "Online DQM Data Stream":http://lariat-dqm.fnal.gov/data-stream?live to see the data block counts and timing for the current run
47 53 Jason St. John
48 53 Jason St. John
h3. More useful web pages
49 53 Jason St. John
50 66 Jason St. John
* "Run Summary":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary of recent runs
51 53 Jason St. John
* "Using ACNET for monitoring":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring?parent=Wiki This page contains the instructions to monitor the beam status through ACNET. 
52 53 Jason St. John
* "Synoptic instructions":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Using_Synoptic_for_monitoring
53 53 Jason St. John
* "MCR Operations Elog":https://www-bd.fnal.gov/Elog/?logNames=Operations +in the new tab+ This is the Main Control Room (MCR) Elog. Use it to check any possible problem affecting the MCenter beam.
54 62 Jason St. John
* "LArIAT Elog":http://dbweb0.fnal.gov/ECL/lariat/E/index Log the previous shifter out (if they are logged in), and log yourself in. Make a new entry with the "Start of Shift Checklist" form.
55 53 Jason St. John
56 1 Jason St. John
h2.  %{color:green}During the shift%
57 16 Luke Batten
58 12 Luke Batten
* If you have questions, call an expert! "LArIAT Call List":https://docs.google.com/spreadsheets/d/1BCeFLgITn6lzCUgPFUd5y_5taIngt-yKgxg8y2l-tNY/edit?usp=sharing
59 16 Luke Batten
* Or, you can use Google Hangouts: send an email to lariat.shift@gmail.com to get invitation.
60 29 Luke Batten
* As well as Google Hangouts, you can Skype an expert! How to access Skype:
61 1 Jason St. John
* On the ftbf_user or lariat account:
62 1 Jason St. John
** "cd ~/Downloads/skype-4.3.0.37"
63 1 Jason St. John
** "skype"
64 4 Luke Batten
* You can find the password in the top right blue box of the whiteboard in the control room.
65 1 Jason St. John
66 1 Jason St. John
67 1 Jason St. John
What beam should I request? The "Run Plan":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Run_Plan lists current and upcoming beam requests, along with the other MCenter activities going on.
68 1 Jason St. John
69 1 Jason St. John
h3. Starting a new run
70 1 Jason St. John
71 6 Luke Batten
* Open in a new tab [[LArIAT DAQ Instructions]] and follow the instructions to start a new run.
72 64 Jason St. John
* The DAQ should make an e-log entry for your automatically. 
73 38 Jason St. John
* Special cases:
74 38 Jason St. John
** *Change/check the DAQ configuration* see "DAQ Configuration Instructions":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/1495ShifterInstructions
75 38 Jason St. John
** *Changing gain or filter times* in the ASICS: check that there was [[Teensy Output]] in the initialization spew from the DAQ.
76 1 Jason St. John
77 1 Jason St. John
h3. During an ongoing run
78 1 Jason St. John
79 1 Jason St. John
* Monitor the beam status using the instructions in the "Using ACNET for monitoring":https://redmine.fnal.gov/redmine/projects/lariat-online/wiki/Using_ACNET_for_monitoring, namely 
80 1 Jason St. John
** the *S60* page of ACNET and 
81 29 Luke Batten
** the profile plots for *MC7WC1* and *MC7WC2*.
82 1 Jason St. John
** From ~ftbf_user/acnet or ~lariat/acnet, run @./LArIATViewer.sh@ to open the Synoptic displays.
83 1 Jason St. John
84 39 Jason St. John
* "Run Status Webpage":http://lariat-wbm.fnal.gov/lariat/run-alert.html to monitor the run status (run number, number of spills, number of triggers, etc)
85 39 Jason St. John
* Note that, on the Run Status Webpage, %{color:red}if the run time appears red%, and the spills stop increasing, the run has come to a stop.  It will also speak an audible alert, once.
86 20 Luke Batten
** In order to get it running again, you can issue the 'lariatReset' command in the DAQ terminal, followed by 'go'.
87 20 Luke Batten
** Sometimes the Run Status page freezes. If this happens, see if your DAQ command line has reset. If so, the run has ended. Follow above instructions.
88 65 Jason St. John
* Did the run number fail to increment?  If the number has reset to a low number such as 1 stop the run and then edit /opt/lariat-online/config/runNumber.dat to correct the run number. Start new run after.
89 3 Roberto Acciarri
* "Online DQM":http://lariat-dqm.fnal.gov/ to check basic data integrity. 
90 53 Jason St. John
** If the rate of MWPCs data blocks during the beam spill is not very close to the data block rates of the other systems, debugging this is urgent. !>GoodMWPCsSmall.gif!
91 37 Jason St. John
*** On the Data Blocks page, monitor the *"Data Block Time Stamps"* graph. During the beam (1-5 s) the MWPC line should follow a similar line shape to the V1740 and V1751 lines, and not be too far below.
92 63 Jason St. John
*** You can isolate plots by hovering over the legend.
93 1 Jason St. John
* Use the event viewer to see TPC events. See "Event Viewer Instructions":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki/Event_viewers . If you +decide to post some interesting event+ in the e-log, +please write the number of the spill/event in the title or content of the entry+. It's much easier to search for them later!
94 1 Jason St. John
* Useful: "Recent run history":http://lariat-wbm.fnal.gov/wbm/servlet/LariatRunSummary , python scripts for making [[DQM plots]]
95 1 Jason St. John
96 60 Jason St. John
h1. Troubleshooting
97 1 Jason St. John
98 57 Jason St. John
Most problems solved: "Troubleshooting Page":https://cdcvs.fnal.gov/redmine/projects/lariat-online/wiki#Troubleshooting doesn't help, "call an expert":https://docs.google.com/spreadsheets/d/1BCeFLgITn6lzCUgPFUd5y_5taIngt-yKgxg8y2l-tNY/edit?usp=sharing
99 1 Jason St. John
100 58 Jason St. John
* If a VME Power Supply has to be turned off in an emergency situation (e.g. problem with TPC PMTs) - turn off the regulate function first, then put the decreasing voltage values in the quick ramp window, go to 0 and then turn off!
101 1 Jason St. John
102 1 Jason St. John
h2.  %{color:green}At the end of the shift%
103 1 Jason St. John
104 11 Luke Batten
* Make an entry in the e-Log using the form "Shift Summary" summing up the relevant information from the shift (make sure to include the total trigger counts for the V1751s and V1740s). For any other information to be posted in the e-Log during the shift use the "default" form. 
105 1 Jason St. John
* IF NO SHIFTER IS TAKING OVER FROM YOU
106 1 Jason St. John
# Call MCR to stop the beam 
107 1 Jason St. John
108 1 Jason St. John
h2. *_Remember to call MCR at the beginning and ending of each shift!!!_*
109 1 Jason St. John
110 1 Jason St. John
h1. *_Remember to call MCR at the beginning and ending of each shift!!!_*