Project

General

Profile

Daily check list » History » Version 76

Kathy Romer, 10/18/2017 05:27 PM

1 23 Marcelle Santos
h1. Run Manager's daily check list
2 1 Marc testing Mengel
3 1 Marc testing Mengel
h2. Before 2pm
4 6 Thomas Diehl
5 76 Kathy Romer
(Things in *bold* definitely need to done every day.) 
6 76 Kathy Romer
>[added by Kathy R Oct 2017 - IMHO, other than the email about 4pm meeting, most of this could be left till 3pm....]
7 72 Kathy Romer
8 72 Kathy Romer
# *Send out an email to des-observers@fnal.gov* to say whether the 4pm meeting is happening or not (this may have already been decided at the previous 4pm meeting, but it is still worth sending out that email).
9 28 Thomas Diehl
# Perform Observer 1 and 2 End Of Night Checklists if the observers didn't do it (see their instructions).
10 19 Thomas Diehl
# Write the Synopsis_Night logbook entry if the observers didn't do it.
11 32 Kathy Romer
# Publish the nightsum in the morning if the observers didn't do it.
12 5 Thomas Diehl
# Complete CTIO end of night report if the observers didn't do it (if you had two operators, make sure both names appear). Check "the calendar":http://www.ctio.noao.edu/noao/calendar-node-field-nr-date to see the status of the night reports.
13 1 Marc testing Mengel
# Be aware of our observers arrival to and departure from the mountain. 
14 72 Kathy Romer
# *Prepare the nightly observing plan page*, including populating Almanac and Weather info (go to the Shifters page ("link for Y5":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DESShiftsYr6) and scroll to - and then click on - today's date.) Top tip: populate the new page with the contents from the previous night, and then edit. That way you won't forget the info to include.
15 1 Marc testing Mengel
# Check the instrument status (read the logbook, talk to people)
16 1 Marc testing Mengel
# Check the DM status
17 17 Klaus Honscheid
# Check SISPI status 
18 61 Kathy Romer
# Check what scripts are needed/present for calibration
19 72 Kathy Romer
# Take a first look at the DESDM data quality "results table":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database Anything unusual should be discussed at the 4pm meeting. Typically you won't accept the results until after the 4pm meeting.
20 72 Kathy Romer
# *Check diskspace*.  Log into one of the observer machines as user "sispi"  (password on white board) and execute the commands below. If any disks are filling up (>90% for observer2, >93% for the pipeline machines) send an email to Klaus and Ann (kh@physics.osu.edu and elliotta@mps.ohio-state.edu)
21 55 Kathy Romer
> added by Kathy R 10/17: I didn't have any success logging in using *su sispi* from observer3 but *ssh -X sispi@observer3* did work 
22 4 Thomas Diehl
<pre>
23 4 Thomas Diehl
setup SISPI
24 8 Stephen Kent
check_diskspace
25 8 Stephen Kent
</pre>
26 4 Thomas Diehl
27 2 Thomas Diehl
h2. Late afternoon
28 1 Marc testing Mengel
29 22 Ann Elliot
# Before the 4pm meeting check with telops if SISPI instance and the GUIs were restarted. You could also do this by checking if they completed a "2 PM Checklist" in the eLog, or looking at the "Uptime" field in the Architect Console GUI.
30 31 Douglas Tucker
# Perform the "aTmCam afternoon check":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/ATmCamInfo
31 59 Kathy Romer
# Attend the "4pm meeting":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/4pm_meeting_ (this is usually chaired by someone at Fermilab, but occasionally you might have to do it)
32 74 Kathy Romer
# Edit the observing plan (you should have already started it, see "before 2pm" list above). Differences in the plan do occur from night to night: these are usually discussed at the 4pm meeting.
33 74 Kathy Romer
# Check that telops have started the afternoon calibration flat fielding scripts (see below for an overview of what they have to do).
34 74 Kathy Romer
# Perform the Late Afternoon Checklist in the logbook (this link to logbook only "works at CTIO":http://system1.ctio.noao.edu:8080/ECL/decam/E/index"). See below, for some of the information that is asked for in the logbook:
35 68 Kathy Romer
# "check the SN processing status":https://desar2.cosmology.illinois.edu/DESFiles/desardata/QA/sn_status.html [for this you need a password that would be the first 3 letters of your Services username + docdbpw (the inaccurate info about how many CCDs) ]
36 68 Kathy Romer
# "Implement the data quality":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database (= "fill out the Survey Table") *If you didn't have a 4pm meeting, then you need to check with DESDM before doing the implementation if you think any of their true/false flags are erroneous.* (Note that it is OK to implement the data quality later in the evening/night if you didn't do it sooner--it won't mess up the data from that night.)
37 68 Kathy Romer
# Make sure observers are present and roles are defined/understood
38 68 Kathy Romer
39 69 Kathy Romer
> Examples of things you will enter into the logbook
40 1 Marc testing Mengel
> * names of  the observers
41 53 Devon Hollowood
> * the "SISPI Instance": The format is DECam_YYYYMMDD
42 68 Kathy Romer
> * name of the script used to perform the afternoon calibrations (most likely this will be prenight_zeros_flats.json)
43 1 Marc testing Mengel
> * confirmation that the morning (i.e. dawn of the day you are in at the moment) calibration has been completed
44 53 Devon Hollowood
> * confirmation that the afternoon (i.e. about now on the day you are in at the moment) calibration has been completed
45 53 Devon Hollowood
46 69 Kathy Romer
> Overview of how calibrations are done (these will be done by Telops, don't attempt them yourself!)
47 70 Kathy Romer
> * Verify Dome is Closed 
48 70 Kathy Romer
> * Verify Dome Flat Setup (ask Telops) [[domeflatsetup|Dome Flat setup]]
49 70 Kathy Romer
> * Turn VSUB on
50 70 Kathy Romer
> * Submit Afternoon Calibration Script (/user/sispi/decam/ExposureScripts/DECam/prenight_zeros_flats.json):
51 68 Kathy Romer
52 69 Kathy Romer
> Overview of how calibrations are checked (this should be done by Observer2, but you might have to do it yourself)
53 70 Kathy Romer
> * Check a few exposures on the Comfort Display. Is the focal plane complete?
54 70 Kathy Romer
> * When script is done, in kentools (on observer2),  perform "flatCheck"
55 5 Thomas Diehl
56 1 Marc testing Mengel
h2. Night
57 1 Marc testing Mengel
58 37 Christopher Bonnett
# Make sure the observers understand what they are doing
59 7 Thomas Diehl
# Make sure the observers get off to a good start
60 36 Christopher Bonnett
# Periodically check wind speed, TelOPS will lower/raise screens if needed. 
61 7 Thomas Diehl
# Be available to the observers if they have questions or problems
62 7 Thomas Diehl
# Try to get some sleep