Project

General

Profile

Daily check list » History » Version 90

Kathy Romer, 10/19/2017 01:45 PM

1 23 Marcelle Santos
h1. Run Manager's daily check list
2 1 Marc testing Mengel
3 1 Marc testing Mengel
h2. Before 2pm
4 6 Thomas Diehl
5 84 Kathy Romer
(Not all of this happens in practice, but things in *bold* definitely need to done every day.) 
6 77 Kathy Romer
>[added by Kathy R 10/17 - IMHO, other than the email about 4pm meeting, most of this could be left till 3pm....]
7 72 Kathy Romer
8 72 Kathy Romer
# *Send out an email to des-observers@fnal.gov* to say whether the 4pm meeting is happening or not (this may have already been decided at the previous 4pm meeting, but it is still worth sending out that email).
9 28 Thomas Diehl
# Perform Observer 1 and 2 End Of Night Checklists if the observers didn't do it (see their instructions).
10 19 Thomas Diehl
# Write the Synopsis_Night logbook entry if the observers didn't do it.
11 32 Kathy Romer
# Publish the nightsum in the morning if the observers didn't do it.
12 5 Thomas Diehl
# Complete CTIO end of night report if the observers didn't do it (if you had two operators, make sure both names appear). Check "the calendar":http://www.ctio.noao.edu/noao/calendar-node-field-nr-date to see the status of the night reports.
13 1 Marc testing Mengel
# Be aware of our observers arrival to and departure from the mountain. 
14 90 Kathy Romer
# *Prepare the nightly observing plan page*, including populating Almanac and Weather info (go to the Shifters page ("link for Y5":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DESShiftsYr6) and scroll to - and then click on - today's date.) Top tip: populate the new page with the contents from the previous night, and then edit. That way you won't forget what  info to include. (Writing the observing plan takes about 15 minutes.)
15 1 Marc testing Mengel
# Check the instrument status (read the logbook, talk to people)
16 1 Marc testing Mengel
# Check the DM status
17 17 Klaus Honscheid
# Check SISPI status 
18 61 Kathy Romer
# Check what scripts are needed/present for calibration
19 72 Kathy Romer
# Take a first look at the DESDM data quality "results table":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database Anything unusual should be discussed at the 4pm meeting. Typically you won't accept the results until after the 4pm meeting.
20 72 Kathy Romer
# *Check diskspace*.  Log into one of the observer machines as user "sispi"  (password on white board) and execute the commands below. If any disks are filling up (>90% for observer2, >93% for the pipeline machines) send an email to Klaus and Ann (kh@physics.osu.edu and elliotta@mps.ohio-state.edu)
21 55 Kathy Romer
> added by Kathy R 10/17: I didn't have any success logging in using *su sispi* from observer3 but *ssh -X sispi@observer3* did work 
22 4 Thomas Diehl
<pre>
23 4 Thomas Diehl
setup SISPI
24 8 Stephen Kent
check_diskspace
25 8 Stephen Kent
</pre>
26 4 Thomas Diehl
27 2 Thomas Diehl
h2. Late afternoon
28 1 Marc testing Mengel
29 22 Ann Elliot
# Before the 4pm meeting check with telops if SISPI instance and the GUIs were restarted. You could also do this by checking if they completed a "2 PM Checklist" in the eLog, or looking at the "Uptime" field in the Architect Console GUI.
30 31 Douglas Tucker
# Perform the "aTmCam afternoon check":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/ATmCamInfo
31 59 Kathy Romer
# Attend the "4pm meeting":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/4pm_meeting_ (this is usually chaired by someone at Fermilab, but occasionally you might have to do it)
32 74 Kathy Romer
# Edit the observing plan (you should have already started it, see "before 2pm" list above). Differences in the plan do occur from night to night: these are usually discussed at the 4pm meeting.
33 74 Kathy Romer
# Check that telops have started the afternoon calibration flat fielding scripts (see below for an overview of what they have to do).
34 83 Kathy Romer
# Perform the Late Afternoon Checklist in the logbook. See below, for some of the information that is asked for in the logbook, and for instructions to get onto that checklist.
35 88 Kathy Romer
# "Implement the data quality":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database (= "fill out the Survey Table") *If you didn't have a 4pm meeting*, then you need to check with DESDM before doing the implementation if you think any of their true/false flags are erroneous. (Note that it is OK to implement the data quality later in the evening/night if you didn't do it in the afternoon.)
36 1 Marc testing Mengel
# Make sure observers are present and roles are defined/understood
37 89 Kathy Romer
# "check the SN processing status":https://desar2.cosmology.illinois.edu/DESFiles/desardata/QA/sn_status.html [for this you need a password that would be the first 3 letters of your Services username + the multiple fried potato password we use with docdb]
38 89 Kathy Romer
>note from Kathy R (10/17) - as far as I can tell this last step is redundant not for RM (it is done by members of the SN WG and reported on at the 4pm meeting)
39 68 Kathy Romer
40 83 Kathy Romer
> Examples of things you will enter into the Late Afternoon Checklist in the logbook
41 1 Marc testing Mengel
> * names of  the observers
42 53 Devon Hollowood
> * the "SISPI Instance": The format is DECam_YYYYMMDD
43 1 Marc testing Mengel
> * name of the script used to perform the afternoon calibrations (most likely this will be prenight_zeros_flats.json)
44 1 Marc testing Mengel
> * confirmation that today's afternoon calibration has been completed
45 83 Kathy Romer
46 83 Kathy Romer
> How to get to Late Afternoon Checklist in the logbook
47 83 Kathy Romer
> *  "Click on this link":http://system1.ctio.noao.edu:8080/ECL/decam/E/index" (N.B. this works at CTIO even from laptops, but not from elsewhere)
48 83 Kathy Romer
> * login as DECamObserver (password on white board)
49 85 Kathy Romer
> * select New Entry from left side bar (see image below, left), and a new window pops up.
50 85 Kathy Romer
> * select "Afternoon Check List" from the Form menu and "DECam Logbook" from the Category menu. Click on "Summary" in Tags window (see image below, right).
51 83 Kathy Romer
> * after you've filled it in, press post (button at the end)
52 53 Devon Hollowood
53 69 Kathy Romer
> Overview of how calibrations are done (these will be done by Telops, don't attempt them yourself!)
54 70 Kathy Romer
> * Verify Dome is Closed 
55 70 Kathy Romer
> * Verify Dome Flat Setup (ask Telops) [[domeflatsetup|Dome Flat setup]]
56 70 Kathy Romer
> * Turn VSUB on
57 70 Kathy Romer
> * Submit Afternoon Calibration Script (/user/sispi/decam/ExposureScripts/DECam/prenight_zeros_flats.json):
58 68 Kathy Romer
59 69 Kathy Romer
> Overview of how calibrations are checked (this should be done by Observer2, but you might have to do it yourself)
60 1 Marc testing Mengel
> * Check a few exposures on the Comfort Display. Is the focal plane complete?
61 85 Kathy Romer
> * When script is done, in kentools (on observer2),  perform "flatCheck"
62 5 Thomas Diehl
63 1 Marc testing Mengel
h2. Night
64 1 Marc testing Mengel
65 37 Christopher Bonnett
# Make sure the observers understand what they are doing
66 7 Thomas Diehl
# Make sure the observers get off to a good start
67 80 Kathy Romer
# Periodically check wind speed (TelOPS will lower/raise screens if needed) 
68 7 Thomas Diehl
# Be available to the observers if they have questions or problems
69 7 Thomas Diehl
# Try to get some sleep
70 87 Kathy Romer
71 87 Kathy Romer
72 87 Kathy Romer
p=. !GettingToPMChecklist.png!