Project

General

Profile

Daily check list » History » Version 73

Kathy Romer, 10/18/2017 05:23 PM

1 23 Marcelle Santos
h1. Run Manager's daily check list
2 1 Marc testing Mengel
3 1 Marc testing Mengel
h2. Before 2pm
4 6 Thomas Diehl
5 72 Kathy Romer
(Things in *bold* definitely need to done every day.)
6 72 Kathy Romer
7 72 Kathy Romer
# *Send out an email to des-observers@fnal.gov* to say whether the 4pm meeting is happening or not (this may have already been decided at the previous 4pm meeting, but it is still worth sending out that email).
8 28 Thomas Diehl
# Perform Observer 1 and 2 End Of Night Checklists if the observers didn't do it (see their instructions).
9 19 Thomas Diehl
# Write the Synopsis_Night logbook entry if the observers didn't do it.
10 32 Kathy Romer
# Publish the nightsum in the morning if the observers didn't do it.
11 5 Thomas Diehl
# Complete CTIO end of night report if the observers didn't do it (if you had two operators, make sure both names appear). Check "the calendar":http://www.ctio.noao.edu/noao/calendar-node-field-nr-date to see the status of the night reports.
12 1 Marc testing Mengel
# Be aware of our observers arrival to and departure from the mountain. 
13 72 Kathy Romer
# *Prepare the nightly observing plan page*, including populating Almanac and Weather info (go to the Shifters page ("link for Y5":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DESShiftsYr6) and scroll to - and then click on - today's date.) Top tip: populate the new page with the contents from the previous night, and then edit. That way you won't forget the info to include.
14 1 Marc testing Mengel
# Check the instrument status (read the logbook, talk to people)
15 1 Marc testing Mengel
# Check the DM status
16 17 Klaus Honscheid
# Check SISPI status 
17 61 Kathy Romer
# Check what scripts are needed/present for calibration
18 72 Kathy Romer
# Take a first look at the DESDM data quality "results table":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database Anything unusual should be discussed at the 4pm meeting. Typically you won't accept the results until after the 4pm meeting.
19 72 Kathy Romer
# *Check diskspace*.  Log into one of the observer machines as user "sispi"  (password on white board) and execute the commands below. If any disks are filling up (>90% for observer2, >93% for the pipeline machines) send an email to Klaus and Ann (kh@physics.osu.edu and elliotta@mps.ohio-state.edu)
20 55 Kathy Romer
> added by Kathy R 10/17: I didn't have any success logging in using *su sispi* from observer3 but *ssh -X sispi@observer3* did work 
21 4 Thomas Diehl
<pre>
22 4 Thomas Diehl
setup SISPI
23 8 Stephen Kent
check_diskspace
24 8 Stephen Kent
</pre>
25 4 Thomas Diehl
26 2 Thomas Diehl
h2. Late afternoon
27 1 Marc testing Mengel
28 22 Ann Elliot
# Before the 4pm meeting check with telops if SISPI instance and the GUIs were restarted. You could also do this by checking if they completed a "2 PM Checklist" in the eLog, or looking at the "Uptime" field in the Architect Console GUI.
29 31 Douglas Tucker
# Perform the "aTmCam afternoon check":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/ATmCamInfo
30 59 Kathy Romer
# Attend the "4pm meeting":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/4pm_meeting_ (this is usually chaired by someone at Fermilab, but occasionally you might have to do it)
31 73 Kathy Romer
# Edit the observing plan (you should have already started it, see above). Differences in the plan do occur from night to night: these are usually discussed at the 4pm meeting.
32 71 Kathy Romer
# Check that telops have started the afternoon calibration flat fielding scripts (see below for an overview of what they have to do)
33 68 Kathy Romer
# Perform the Late Afternoon Checklist in the logbook. (link to logbook that only "works at CTIO":http://system1.ctio.noao.edu:8080/ECL/decam/E/index") See below, for some of the information that is asked for in the logbook:
34 68 Kathy Romer
# "check the SN processing status":https://desar2.cosmology.illinois.edu/DESFiles/desardata/QA/sn_status.html [for this you need a password that would be the first 3 letters of your Services username + docdbpw (the inaccurate info about how many CCDs) ]
35 68 Kathy Romer
# "Implement the data quality":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database (= "fill out the Survey Table") *If you didn't have a 4pm meeting, then you need to check with DESDM before doing the implementation if you think any of their true/false flags are erroneous.* (Note that it is OK to implement the data quality later in the evening/night if you didn't do it sooner--it won't mess up the data from that night.)
36 68 Kathy Romer
# Make sure observers are present and roles are defined/understood
37 68 Kathy Romer
38 69 Kathy Romer
> Examples of things you will enter into the logbook
39 1 Marc testing Mengel
> * names of  the observers
40 53 Devon Hollowood
> * the "SISPI Instance": The format is DECam_YYYYMMDD
41 68 Kathy Romer
> * name of the script used to perform the afternoon calibrations (most likely this will be prenight_zeros_flats.json)
42 1 Marc testing Mengel
> * confirmation that the morning (i.e. dawn of the day you are in at the moment) calibration has been completed
43 53 Devon Hollowood
> * confirmation that the afternoon (i.e. about now on the day you are in at the moment) calibration has been completed
44 53 Devon Hollowood
45 69 Kathy Romer
> Overview of how calibrations are done (these will be done by Telops, don't attempt them yourself!)
46 70 Kathy Romer
> * Verify Dome is Closed 
47 70 Kathy Romer
> * Verify Dome Flat Setup (ask Telops) [[domeflatsetup|Dome Flat setup]]
48 70 Kathy Romer
> * Turn VSUB on
49 70 Kathy Romer
> * Submit Afternoon Calibration Script (/user/sispi/decam/ExposureScripts/DECam/prenight_zeros_flats.json):
50 68 Kathy Romer
51 69 Kathy Romer
> Overview of how calibrations are checked (this should be done by Observer2, but you might have to do it yourself)
52 70 Kathy Romer
> * Check a few exposures on the Comfort Display. Is the focal plane complete?
53 70 Kathy Romer
> * When script is done, in kentools (on observer2),  perform "flatCheck"
54 5 Thomas Diehl
55 1 Marc testing Mengel
h2. Night
56 1 Marc testing Mengel
57 37 Christopher Bonnett
# Make sure the observers understand what they are doing
58 7 Thomas Diehl
# Make sure the observers get off to a good start
59 36 Christopher Bonnett
# Periodically check wind speed, TelOPS will lower/raise screens if needed. 
60 7 Thomas Diehl
# Be available to the observers if they have questions or problems
61 7 Thomas Diehl
# Try to get some sleep