Project

General

Profile

Daily check list » History » Version 81

Version 80 (Kathy Romer, 10/18/2017 05:30 PM) → Version 81/96 (Kathy Romer, 10/18/2017 05:33 PM)

h1. Run Manager's daily check list

h2. Before 2pm

(Things in *bold* definitely need to done every day.)
>[added by Kathy R 10/17 - IMHO, other than the email about 4pm meeting, most of this could be left till 3pm....]

# *Send out an email to des-observers@fnal.gov* to say whether the 4pm meeting is happening or not (this may have already been decided at the previous 4pm meeting, but it is still worth sending out that email).
# Perform Observer 1 and 2 End Of Night Checklists if the observers didn't do it (see their instructions).
# Write the Synopsis_Night logbook entry if the observers didn't do it.
# Publish the nightsum in the morning if the observers didn't do it.
# Complete CTIO end of night report if the observers didn't do it (if you had two operators, make sure both names appear). Check "the calendar":http://www.ctio.noao.edu/noao/calendar-node-field-nr-date to see the status of the night reports.
# Be aware of our observers arrival to and departure from the mountain.
# *Prepare the nightly observing plan page*, including populating Almanac and Weather info (go to the Shifters page ("link for Y5":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DESShiftsYr6) and scroll to - and then click on - today's date.) Top tip: populate the new page with the contents from the previous night, and then edit. That way you won't forget the info to include.
# Check the instrument status (read the logbook, talk to people)
# Check the DM status
# Check SISPI status
# Check what scripts are needed/present for calibration
# Take a first look at the DESDM data quality "results table":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database Anything unusual should be discussed at the 4pm meeting. Typically you won't accept the results until after the 4pm meeting.
# *Check diskspace*. Log into one of the observer machines as user "sispi" (password on white board) and execute the commands below. If any disks are filling up (>90% for observer2, >93% for the pipeline machines) send an email to Klaus and Ann (kh@physics.osu.edu and elliotta@mps.ohio-state.edu)
> added by Kathy R 10/17: I didn't have any success logging in using *su sispi* from observer3 but *ssh -X sispi@observer3* did work
<pre>
setup SISPI
check_diskspace
</pre>

h2. Late afternoon

# Before the 4pm meeting check with telops if SISPI instance and the GUIs were restarted. You could also do this by checking if they completed a "2 PM Checklist" in the eLog, or looking at the "Uptime" field in the Architect Console GUI.
# Perform the "aTmCam afternoon check":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/ATmCamInfo
# Attend the "4pm meeting":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/4pm_meeting_ (this is usually chaired by someone at Fermilab, but occasionally you might have to do it)
# Edit the observing plan (you should have already started it, see "before 2pm" list above). Differences in the plan do occur from night to night: these are usually discussed at the 4pm meeting.
# Check that telops have started the afternoon calibration flat fielding scripts (see below for an overview of what they have to do).
# Perform the Late Afternoon Checklist in the logbook (this link to logbook "that works at CTIO":http://system1.ctio.noao.edu:8080/ECL/decam/E/index"). See below, for some of the information that is asked for in the logbook:
# "check the SN processing status":https://desar2.cosmology.illinois.edu/DESFiles/desardata/QA/sn_status.html [for this you need a password that would be the first 3 letters of your Services username + docdbpw (the inaccurate info about how many CCDs) ]
# "Implement the data quality":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database (= "fill out the Survey Table") *If you didn't have a 4pm meeting, then you need to check with DESDM before doing the implementation if you think any of their true/false flags are erroneous.* (Note that it is OK to implement the data quality later in the evening/night if you didn't do it in sooner--it won't mess up the afternoon.) data from that night.)
# Make sure observers are present and roles are defined/understood

> Examples of things you will enter into the logbook
> * names of the observers
> * the "SISPI Instance": The format is DECam_YYYYMMDD
> * name of the script used to perform the afternoon calibrations (most likely this will be prenight_zeros_flats.json)
> * confirmation that today's afternoon calibration has been completed

> Overview of how calibrations are done (these will be done by Telops, don't attempt them yourself!)
> * Verify Dome is Closed
> * Verify Dome Flat Setup (ask Telops) [[domeflatsetup|Dome Flat setup]]
> * Turn VSUB on
> * Submit Afternoon Calibration Script (/user/sispi/decam/ExposureScripts/DECam/prenight_zeros_flats.json):

> Overview of how calibrations are checked (this should be done by Observer2, but you might have to do it yourself)
> * Check a few exposures on the Comfort Display. Is the focal plane complete?
> * When script is done, in kentools (on observer2), perform "flatCheck"

h2. Night

# Make sure the observers understand what they are doing
# Make sure the observers get off to a good start
# Periodically check wind speed (TelOPS will lower/raise screens if needed)
# Be available to the observers if they have questions or problems
# Try to get some sleep