Daily check list » History » Version 87
Kathy Romer, 10/18/2017 07:03 PM
h1. Run Manager's daily check list
|2||1||Marc testing Mengel|
|3||1||Marc testing Mengel||
h2. Before 2pm
(Not all of this happens in practice, but things in *bold* definitely need to done every day.)
>[added by Kathy R 10/17 - IMHO, other than the email about 4pm meeting, most of this could be left till 3pm....]
# *Send out an email to firstname.lastname@example.org* to say whether the 4pm meeting is happening or not (this may have already been decided at the previous 4pm meeting, but it is still worth sending out that email).
# Perform Observer 1 and 2 End Of Night Checklists if the observers didn't do it (see their instructions).
# Write the Synopsis_Night logbook entry if the observers didn't do it.
# Publish the nightsum in the morning if the observers didn't do it.
# Complete CTIO end of night report if the observers didn't do it (if you had two operators, make sure both names appear). Check "the calendar":http://www.ctio.noao.edu/noao/calendar-node-field-nr-date to see the status of the night reports.
|13||1||Marc testing Mengel||
# Be aware of our observers arrival to and departure from the mountain.
# *Prepare the nightly observing plan page*, including populating Almanac and Weather info (go to the Shifters page ("link for Y5":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DESShiftsYr6) and scroll to - and then click on - today's date.) Top tip: populate the new page with the contents from the previous night, and then edit. That way you won't forget the info to include.
|15||1||Marc testing Mengel||
# Check the instrument status (read the logbook, talk to people)
|16||1||Marc testing Mengel||
# Check the DM status
# Check SISPI status
# Check what scripts are needed/present for calibration
# Take a first look at the DESDM data quality "results table":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database Anything unusual should be discussed at the 4pm meeting. Typically you won't accept the results until after the 4pm meeting.
# *Check diskspace*. Log into one of the observer machines as user "sispi" (password on white board) and execute the commands below. If any disks are filling up (>90% for observer2, >93% for the pipeline machines) send an email to Klaus and Ann (email@example.com and firstname.lastname@example.org)
> added by Kathy R 10/17: I didn't have any success logging in using *su sispi* from observer3 but *ssh -X sispi@observer3* did work
h2. Late afternoon
|28||1||Marc testing Mengel|
# Before the 4pm meeting check with telops if SISPI instance and the GUIs were restarted. You could also do this by checking if they completed a "2 PM Checklist" in the eLog, or looking at the "Uptime" field in the Architect Console GUI.
# Perform the "aTmCam afternoon check":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/ATmCamInfo
# Attend the "4pm meeting":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/4pm_meeting_ (this is usually chaired by someone at Fermilab, but occasionally you might have to do it)
# Edit the observing plan (you should have already started it, see "before 2pm" list above). Differences in the plan do occur from night to night: these are usually discussed at the 4pm meeting.
# Check that telops have started the afternoon calibration flat fielding scripts (see below for an overview of what they have to do).
# Perform the Late Afternoon Checklist in the logbook. See below, for some of the information that is asked for in the logbook, and for instructions to get onto that checklist.
# "check the SN processing status":https://desar2.cosmology.illinois.edu/DESFiles/desardata/QA/sn_status.html [for this you need a password that would be the first 3 letters of your Services username + docdbpw (the inaccurate info about how many CCDs) ]
# "Implement the data quality":https://cdcvs.fnal.gov/redmine/projects/desops/wiki/DM_into_Database (= "fill out the Survey Table") *If you didn't have a 4pm meeting, then you need to check with DESDM before doing the implementation if you think any of their true/false flags are erroneous.* (Note that it is OK to implement the data quality later in the evening/night if you didn't do it in the afternoon.)
# Make sure observers are present and roles are defined/understood
> Examples of things you will enter into the Late Afternoon Checklist in the logbook
|40||1||Marc testing Mengel||
> * names of the observers
> * the "SISPI Instance": The format is DECam_YYYYMMDD
|42||1||Marc testing Mengel||
> * name of the script used to perform the afternoon calibrations (most likely this will be prenight_zeros_flats.json)
|43||1||Marc testing Mengel||
> * confirmation that today's afternoon calibration has been completed
> How to get to Late Afternoon Checklist in the logbook
> * "Click on this link":http://system1.ctio.noao.edu:8080/ECL/decam/E/index" (N.B. this works at CTIO even from laptops, but not from elsewhere)
> * login as DECamObserver (password on white board)
> * select New Entry from left side bar (see image below, left), and a new window pops up.
> * select "Afternoon Check List" from the Form menu and "DECam Logbook" from the Category menu. Click on "Summary" in Tags window (see image below, right).
> * after you've filled it in, press post (button at the end)
> Overview of how calibrations are done (these will be done by Telops, don't attempt them yourself!)
> * Verify Dome is Closed
> * Verify Dome Flat Setup (ask Telops) [[domeflatsetup|Dome Flat setup]]
> * Turn VSUB on
> * Submit Afternoon Calibration Script (/user/sispi/decam/ExposureScripts/DECam/prenight_zeros_flats.json):
> Overview of how calibrations are checked (this should be done by Observer2, but you might have to do it yourself)
|59||1||Marc testing Mengel||
> * Check a few exposures on the Comfort Display. Is the focal plane complete?
> * When script is done, in kentools (on observer2), perform "flatCheck"
|63||1||Marc testing Mengel||
|64||1||Marc testing Mengel|
# Make sure the observers understand what they are doing
# Make sure the observers get off to a good start
# Periodically check wind speed (TelOPS will lower/raise screens if needed)
# Be available to the observers if they have questions or problems
# Try to get some sleep