Punch List


Telescope / Software state changes

  • 31 October: Was pointing map from 8 Oct observations installed on this night?
  • 5 November: Pointing map from 31 Oct observations installed this night?
  • 5 November: Guider changes, including new code to avoid choosing CRs as guide stars, delay guider readout until after shutter is all the way open, and change timing so that guider CCDs are not cleared while previous exposure is still reading out on science array. Latter change seems to have eliminated the readout striping issue, and former 2 have greatly reduced incidence of guider failing to find stars.
  • 6 November: Commissioning pointing map (re-)installed. Much improved track. Somewhat worse pointing.
  • 7 November: TCS software modifications: updated pointing model (not clear if based on old mosaic or newer models). Telescope response to 1" "kicks" now decoupled in RA, dec. Problem with response in RA (too large - now in both E and W directions)
  • 8 November: TCS modification to allow pointing S of -79 dec.
  • 11 Nov: correct telescope LUT installed at image # 150014
  • 15 Nov: Rasicam mirrors cleaned. OBSTAC given list of Minisurvey exposures to ignore (MS1a izY).
  • 19 Nov: OBSTAC max airmass is 1.5 and no longer ignoring the MS1a izY's.
  • 19 Nov: Change to SISPI so that BCAM is not recalibrated on every Configure operation.
  • 20 Nov: Black felt applied to parts of the telescope.
  • 22 Nov: BCAM analysis matrix changed, to align BCAM outputs with hexapod axis & center of rotation.
  • 24 Nov: New hexapod LUT installed by Kevin.
  • 25-30 Nov: Several changes to tracking, new pointing model, "RA damper motor" plugged in.
  • 3 Dec, 4 Dec: New guider algorithms tested early in night and then reverted.
  • 3 Dec (and 4 Dec?): bad dome encoder, lots of occulted exposures. Then repaired.
  • 5 Dec: New guider algorithm installed (pyguide)
  • 6 Dec: Primary mirror washed.
  • 6 Dec: New filter focus LUT installed. Change to guider timing & feedback parameters (guider latency set to 0, 10% feedback)
  • 11 Dec: Ion pump moved.
  • 17 Dec: Mamac replaced on telescope primary support system- #22
  • 17 Dec: Work on RA cable wraps
  • 19 Dec: TCS tweak to fix accuracy of small offsets (German S.). Telescope rebalanced. Units of guider logs changed and now gives error signal instead of correction signal.
  • 29 Dec - 14 Jan: Environmental database never came back up after the power outages, so temperatures etc. over this period are invalid.
  • Jan: ongoing work on the RA cable wrap.
  • 23-28 Jan: TCS, damper motor, cable wrap work during engineering.
  • Jan 28 ???: begin routine 5-axis AOS control of hexapod during DES halves of nights.
  • 30 Jan: Fixed the oil bearings on horseshoe, tracking friction greatly reduced.
  • 1 Feb: New chiller online, begin using for mirror and dome floor cooling. Consistent daytime cooling starts 4 Feb (?)
  • 1 Feb: Rasicam now properly receiving position of Blanco pointing.
  • 17 Feb: Image Health PSF outputs changed (table imagehealth_fp). Note change of units and some other things in following days.
  • 25 Mar: Guider configuration changed (eLog 67424). Smaller isolation range, search range and higher tolerance to missing guide stars
  • 25 Mar: New pointing model installed and a fix to the TCS offset code (cos(dec) term?)
  • 27 Mar: TCS Delay reduced from 5 seconds to 1 second
  • 27 Mar: Default AOS mode is now aos5 (all 5 degrees of freedom in closed loop)

An incomplete list for Year 1

  • 29 October Rasicam flat field updated. It was cleaned a few weeks ago…
  • 6 November Rescaled IH algorithm to calculate seeing to match QC values.
  • 6 November Modified obstac query of IH data use only exposures with ih_done and ih_accept both marked true in the exposure database
  • 6 November Change the filter seeing correction table used by obstac to correct to i band instead of DIMM.
  • 11 November: excess noise becomes visible in the bias frames, probably present until sync cable fix on 30 Nov
  • 12 November Chiller failure caused ~3K focal plane temperature excursion around 07:00 UT
  • c. 15 November Repair dome floor cooling system temperature sensor, and change daytime mirror-cooling set point algorithm.
  • 26 November: Hexapod differential encoder failure, switched to encoder on the motor.
  • 30 November: Bad sync cable in readouts, power supply replaced, S30 dies.
  • 5 December: Mirror washed
  • 6 December: Small spike in vacuum pressure during manipulation of LN2 lines.
  • 18 December: External Photodiode/Dome light protection goes online
  • 18 December: New initial ztrim equation: 1464 - 122*upper_truss_temperature (instead of 1400 - 110*T)
  • 14 January: Switched back to original Hexapod encodes ("command 11" was removed)
  • 14 January: Oil cooler in operation.
  • 21 January: Focal plane temperature glitch.
  • 30 January: Change voltage on the bluest LED set.
  • 11 March: Changed default constants: new gains, noise values, new filter constants for VR filter (DECAM:54)
  • 11 March: Changed DEC guider feedback from 10% to 5%
  • 27 March: Removed BCAMs from default exposure sequence (now only called for zeros and dome flats)
  • April: Sometime in April the TCS delay was set back to 3 s.
  • 16 May: TCS Delay (3s) coupled to telescope slew, decoupled from dome movement (should the dome be last there won't be an extra delay)
  • 17 May: Fast readout mode set as default
  • 31 July: Changed ImageHealth to exclude dead CCDs (S30, N30, S7) and raised glowing edge cut from 15 to 25

As of 14 Aug 2014 this list is being maintained at
Make any further additions at that page.

Task List

Analyses and jobs that need to be done or problems needing attention:

  1. Make sure that RASICAM data are getting into image headers (Kevin)
  2. Correlate RASICAM image statistics with photometric behavior. Prerequisite: relative photometric solutions for SN, SPT observations, or at least for all standards fields. (Bob, Kevin)
  3. Check if/why the 20s r-band flats with saturation in corners were not getting flagged or rejected in DESDM flat production.
  4. DECal: synthesize maps of variation of stellar color corrections across the array; agreement with star-flat solutions? (Bob, Douglas)
  5. DECal: Is there frequency dependence of the i-band "mountain" that sheds light on its origin? (Wester + ??)
  6. DECal: Make this usable by observers on cloudy nights!!! (Darren ???)
  7. Debug negative values in darktime header field (e.g. 167089) (Klaus ??)
  8. Tree rings and other astrometric distortions: characterize and see that they are handled correctly in flat-fielding and photometry (Gary, Andres)
  9. Do tree rings and glowing edges scale with filter as expected? (Andres)
  10. Update SISPI baseline astrometric solutions (Gary)
    1. With improved telescope pointing revisit the starfinder (and guider auto mode) (Klaus, Eric)
    2. With improved telescope pointing revisit the idea to dynamically set the WCS keywords in the FITS header (Gary, Eric, Steve)
  11. Devise summary window for SISPI that satisfies cranky old observers like Bob & Gary
  12. Set thresholds for star counts in QR vs exposure time as transparency warnings; likewise at DESDM for entry to Survey Table (Angelo)
  13. Determine intra-night variation of standards-fields zeropoints and set standards policy for 2013 season (Douglas)
  14. Correlate IQ and whisker with wind records Prerequisite: updated IQ data table and fix wind direction sensor.
  15. Investigate the g-band seeing anomaly: does degree of seeing degradation in g depend on other quantities, e.g. airmass? Is atmospheric dispersion a factor?
  16. Integrate robust half-light radius and whisker measures into Image Health for continuous off-season monitoring, validate against offline analyses (Ken, Klaus)
  17. Observing-efficiency statistics: define, measure from this run, automate for next run (Tom, Eric)
  18. Compress TCS data stream into a few values diagnostic of tracking errors above the guider bandwidth (Kevin)
  19. Investigate timing of telescope settling vs shutter-open vs slew distance, using TCS records - will require working with Kevin on the TCS records)
    TCS Studies
  20. Devise DECam regression test suite: observations to conduct in day and night time to check for calibration changes and performance degradations on regular basis. Also applies to TCS performance statistics. (Alistair has conducted a TCS regression set last week; Paul, Daniel have some routines for daytime calibrations)
    1. Prereq for above: implement HA/Dec or Alt/Az scripting with tracking on via SISPI (Klaus)
  21. Understanding the whiskers: determine whether an optical or a tracking phenomenon is dominating current PSF asymmetries. Solutions? (Aaron and new SLAC student Chris Davis)
  22. PSF dependence on counts? careful examination of whether this is present (after correction for known nonlinearities). (Vinu, Nicolas Regnault)
  23. Star flats: generate and understand star flats, implement corrections to data in DESDM (Douglas, Gary, Vinu, Nicolas Regnault, Anne Bauer, Bob) See also the Open Issue below that Y-band star flat from SDSS fields shows anomalous CCD.
  24. Add code to extract image data from F&A chips and devise a mechanism to transfer the data (to SLAC). This will be required for the donut/image quality analysis and can also be used to track pointing precision (Klaus, Aaron, Kevin).
  25. Add feature to exposure scripts to enter HA, Az, El without having the TCS turn off tracking (Klaus)
  26. CCD anomalies: these are all anomalies/problems with the devices that must be understood and either eliminated or calibrated. See "Open Issues" below.
    1. PSF dependence on counts: careful examination of whether this is present (after correction for known nonlinearities).
    2. Low-level nonlinearities on some amps. Are these consistent with single-pixel effects, correctable with lookup tables? (Huan)
    3. High-light-level nonlinearities on all amps: same questions, does a lookup table fix this or is it non-local? (Huan)
    4. Edge bleeds: can these be ameliorated with sequencer change?
    5. Flat-field stability: what is time/length scale of changes in flats, esp. after the warmup (see "Tape Bump" issue below) (Paul already has some results on Tape Bumps)
    6. Non-linear crosstalk (Kerstin; activity started at SiDet with Juan, Tom, new tech coming)
  27. Refine the AOS loop, including BCAM information (Aaron)
  28. Track down the source of "filter edge ghost" (Steve) Done - will need to re-observe after installation of baffles.
  29. Develop nighttime temperature predictor (Niall, Jim)
  30. Improve identification of bad pixels and "funky columns" to be masked. (Peter, Daniel???)
  31. Make sure that all catalog-level SV reqs/goals are on someone's issue list or Acceptance Test.
  32. Investigate reasons for delays for modest slews and examine distribution of slew distances in simulations. (Eric, ???)
  33. Look at guide star relative motion vs separation, perhaps diagnostic of source of seeing contributions.

We have not yet collected an issue list for completeness, S/G separation, other issues that address performance of the DESDM codes. Anyone should feel free to start such a list here!

Open Issues

Hardware/software anomalies not yet understood and solved:

Aberrant CCD in Y-band star flats

Vinu's processing of the Y-band photometric residuals in SDSS field produces one CCD (#5) that is discrepant from its neighbors in the star-to-dome response ratio. Douglas shows that its color term does not appear much different from the other CCDs. Is this just too few Y-band standards? Await results of internal Y-band star flats.

Tape bump changes

Flat field around the tape bumps appears to have changed after the camera warm up / cool down in late December. Would like to understand why this is and see if it is persistent. Unresolved

Here is an update on this:


Severe vertical trailing of images and other oddities on night of 7 Nov. Unresolved

N30 Analysis

Guider problems

Trailing, usually diagonal, in majority of long-exposure images, usually or always accompanied by flags indicating guider fails to find stars.

Commenced 1st night of SV, 1 Nov, continues 2 Nov, critical - long exposures are not possible. See observing logs for these nights and eyeball results.

As of 5 Nov with Mosaic-era pointing map reinstalled, and new guider codes in place, the problem of failure to find guide stars appears resolved and the open-loop tracking errors are reduced from ~1"/min to ~0.1"/min. But images remain substantially elliptical, oscillations are seen in guider GUI. Significant improvements when the old RA damping motor was rediscovered. What has been dubbed guider oscillations are still visible when observing in the north.

Ghost images

Obvious ghosts, showing partial images of the telescope pupil. These are (so far) rare enough to invalidate only a small fraction of the survey area, but they will cause trouble for SExtractor etc and it would be good if an expert could look at these and decide if the ghosts are from some correctable reflection.

Inventory (see the eyeball squad reports):
  • 2 Nov, Exposures 145878, 184882, 145893, 145898 (S14 & S20),

Status: Steve Kent reports:

Say hello to Alpha Phoenicis (constellation Phoenix).  It is a 2nd mag star (K0 giant).  It look like it is a double-bounce inside C5.  
My guess is that the in-focus image is just outside the radius of the guide/FA CCDs.  The baffling is clearly cutting off about
half the beam already. I think that that is about what we expect.  (Not sure if the intensity is what we expect.)  

So we consider this issue resolved except it would be nice to confirm the expected intensity.

Additional efforts by Steve suggest expected ghost brightnesses. Link???


Non-linear response at low light levels in several amps. Now (Jan 2013) have signs of both high- and low-light-level nonlinearities. Refer to A/B Ratio Variation unresolved and critical

Scattered light in flats

Unanticipated gradients in flat fields. Refer to Flat-field Structure unresolved and critical

Charge Pileup (?) at ends of CCDs

CCDs with bright stars on/near them show large triangular regions along serial registers that are obliterated with extra charge. Unresolved although not critical since it affects a small fraction of the survey area.

Look at Exposures page for more info. In fact this behavior appears to occur on all or nearly all CCDs whenever a bleed reaches the end of the device.

The charge pileup is at the serial-register end. Alistair suggests clocking & dumping serial register during readouts.

Nonlinear crosstalk

Discovered by Kerstin Paech. At high ADU levels, crosstalk does not behave as linear function of source ADU level.

Unresolved - not critical for survey operations but should be examined for a software correction. See Nonlinear Crosstalk

Funky columns

Peter spotted columns with slightly higher or lower counts than average in the SV cluster data. Closer inspections indicates that the columns come in pairs, sometime in groups, of warmer and colder columns that seem to compensate each other. They are visible only in fairly low illumination (such a the g- and r-band data of the SV clusters or short PTC exposures), but not in the bias or flat frames. See Funky columns

Closed Issues

Excess noise bands

~100-row stretches with large diagonal bands from excess coherent noise. Appears on CCDs sharing crate with guider chips?
For many example exposures, look at eyeball examination results

Noted occasionally during Commissioning but extensive examples seen starting 1 Nov at SV.

Resolved: was due to guider CCD clearing for following exposure before end of readout. Disappeared after change of timing before Nov 5 observing.

Header alteration with trimming

Several image header values need to be changed when the image is trimmed in order to stay usefully accurate after trimming. These include CRPIX[12] which specify the pixel locations on the sky, DATASEC and CCDSEC which describe the valid data areas of the image, LTV[12] which specify the offset of the pixel indices from the physical pixels on the device. There may be more - an expert on the meaning of the *SEC and the various IRAF coordinate systems should take a look.

Entered 24 Sep 2012 by GaryB; resolved in SISPI and Pipeline during October.

Tracking jumps

Double images in exposures from apparent jumps in tracking. Seen during commissioning, 1-2 images per night on first few SV nights.

Examples include 146952
  • Nov 5: 147536
  • Nov 7: 148556
  • Nov 14: 151287, 151337
  • Search the Exposures page for "guider jump" to get more examples.

fixed (was caused by a problem in the primary mirror support (MAMAC))

Dec limit -79 degrees

Cannot point S of dec=-79, noted Nov 4.

solved (fix put into TCS)