AB ratio variation

Eric Suchyta noted 19 Oct that images show substantial jump at A/B border in chip 31 even after debias and flat-fielding with dome flat. This was traced to fact that these data were flattened using low-level dome flats (~300 ADU - LED's might not even have been on!) and further analysis reveals that the A/B ratio after overscan subtraction depends significantly on illumination level for ~10 of the amplifiers.

Comparison of nonlinearity between 20140729 PTC and fiducial 20130108 PTC, 14 Aug 2014

The nonlinearity results from the recent 20140729 PTC look consistent with those from the fiducial 20130108 PTC, as determined by visual inspection of plots comparing results CCD-by-CCD. Exceptions (as expected) are the unstable right amp of CCD 31, and the now-defunct CCD 2. Plots are attached for the usual example CCDs: 46 (normal), 15 (low-light nonlinear), and 31 (unstable):

Note that the 20140729 PTC shows larger noise due to drifts, time varying scattered light in the dome, etc., cf. the fiducial 20130108 sequence. This is often the case, as the fiducial 20130108 PTC is cleaner than typical.

v0.5 of the nonlinearity lookup table, 04 Sep 2013

Due to artifacts seen in the SV-A1 coadds along the dividing line between amps on CCD 31, it seems that the v0.4 nonlinearity corrections for CCD 31 are doing more harm than good. So I've now removed the CCD 31 nonlinearity corrections, by setting ADU_LINEAR_A = ADU = ADU_LINEAR_B in the attached v0.5 lookup table, linearity_table_v0.5.fits.gz. Hopefully this will reduce the artifacts on CCD 31 to the smaller level seen in the Y1C* coadds for which nonlinearity corrections were not applied.

v0.4 of the nonlinearity lookup table and of the SATURATA,B values table, 21 Jun 2013

I checked into whether the CCD 31 lookup table can be improved, but it looks like from the PTC data from different dates that CCD 31 amp A varies enough that I probably can't do better than the current table. I did fix a bug, specific to CCD 31 amp A, where there was an NaN in the ADU_LINEAR_A value corresponding to ADU = 0. This is now fixed to set ADU_LINEAR_A = 0 there, the same as for all other amps. The updated lookup table is v0.4, linearity_table_v0.4.fits.gz, which should be used for further DESDM processing.

Also, I've attached an ascii file listing the corresponding saturation values, SATURATA and SATURATB, that go along with the nonlinearity corrections (as described before in the 23 May 2013 wiki entry), here: saturatab_v0.4.txt. The same values are also given in the individual extension headers of the lookup table binary fits file.

Check of nonlinearity correction in updated DESDM processing, 12 Jun 2013

I checked the ratios of the median values of the left to right amps in 2 exposures processed by DESDM, w/o and with the v0.3 nonlinearity correction, the former from the Y1C1 release (runs 20130225212202_20121207, 20130225200732_20121124) and the latter from the latest summer reprocessing (runs 20130610164314_20121207, 20130609221352_20121124). The following plots show the left/right ratios vs. CCD number:
  • RXJ2248 g-band exposure 158982: checkamps_158982.png
    • These data have sky background levels about 150-200 ADU, and are in the low-level nonlinearity regime, as indicated by the handful of points that deviate noticeably from 1 in the original, uncorrected data (black).
    • The post-correction ratios (red) look closer to 1, except for the known noisy CCD 31.
  • RXJ2248 z-band exposure 155250: checkamps_155250.png
    • These data have sky levels about 2500-3000 ADU, and are in a regime where there is much less noticeable nonlinearity between left and right amps.
    • The uncorrected and corrected points are nearly the same, except again CCD 31 shows the biggest change.
    • However, though the plotted points are very similar, for some CCDs by eye in ds9 you can tell that there was a difference between the amps that has now gone away due to the nonlinearity correction; see zband_ccd_15_comparison.png for CCD 15. Left is w/o correction, where you can see a left/right difference, while right is with correction, where the difference is no longer visible.

The corrections appear to work well, except for CCD 31. Probably will make an updated lookup table where there is no correction for CCD 31, as applying a correction doesn't seem to make an improvement, presumably due to changes in the temporally unstable CCD 31.

-- Huan Lin

First look at high-level stellar linearity and corrections, 05 Jun 2013

  • Checked stellar photometry linearity using g-band Omega Cen linearity data from 20130221.
  • Comparison of 80-sec exposure (180647) with 10-sec exposure (180638) shown in this plot: plotcheckstars_omegacen.png
    • Left panel is uncorrected fractional nonlinearity, right is corrected using v0.3 linearity lookup table.
    • Nonlinearity at high counts is indeed seen now, and the correction works reasonably well, though probably a bit overcorrected. Should check if there are errors in my sky estimate that could affect low-count end, as well as check against more of the 10-sec images.
    • IDL-measured counts in 11x11 pixel boxes are used. Similar results seen using SExtractor aperture magnitudes, but linearity corrections can be more easily applied to the IDL counts, without having to rerun the image processing.
    • Data from all 62 CCDs (except 31 and 61) are combined to boost S/N.
    • Count rates between 80-sec and 10-sec exposures are normalized to one, independently for each CCD, for objects with average counts per pixel (i.e., the x-axis in the plot) between 500-1000 ADU on the 80-sec image. This improves the tightness of the plotted results (though much more so for smaller measurement apertures, not shown), presumably by correcting for chip-to-chip seeing or PSF differences.
  • Something noticed when trying this with smaller measurements apertures: I can see evidence of the brighter-fatter effect, which operates in the opposite sense to the nonlinearity: for brighter objects, brighter-fatter decreases the counts in a fixed, small aperture since the PSF is now wider, while the nonlinearity increases the counts in the same aperture. To isolate the nonlinearity effect, I needed to make the measurement aperture larger, such as the chosen 11x11 pixel box, or a similar 3-arcsec diameter SExtractor circular aperture.

v0.2 v0.3 of the nonlinearity lookup table, 06 May 2013 23 May 2013

The v0.2 file accidentally had the nonlinearity correction tabulated in the flipped sense, i.e., applying the correction amounted to exactly applying the observed nonlinearity. This is now fixed in v0.3 of the table, given here

I've generated a first version of the nonlinearity lookup table, here

There are 62 binary fits table extensions, one for each CCD (in order; but also given by CCDNUM header keyword in each extension). In each extension, there are 3 columns:
  • ADU: observed ADU, with 65536 values going from 0 to 65535, with a constant step size of 1 ADU
  • ADU_LINEAR_A: linearized ADU for amp A
  • ADU_LINEAR_B: linearized ADU for amp B

To make the nonlinearity correction, use the ADU_LINEAR_[AB] value corresponding to the observed ADU value, interpolating linearly between the table values as needed. The observed ADU value is intended to be the counts measured immediately after subtraction of both overscan and bias. The left vs. right senses of the A and B amps here are consistent with their definitions in the DECam image headers.

Some notes:
  • The tables were constructed by stitching together and linearly interpolating the following 3 data sets:
    • r-band PTC data from 20130108 for >~ 100 ADU
    • r-band "fiber flat" data from 20131025, for low counts ~10-100 ADU
    • r-band sky background data from on-sky exposures of the BD+17 field from 20121106, for very low counts <~ 10 ADU
  • The 20130108 data are very good quality (see further down this wiki page), while the other 2 data sets may suffer from drifts (not correctable as there were no reference exposures) and thus could be replaced by future, improved data.
  • The ADU_LINEAR_[AB] values were forced to be 0 at ADU = 0.
  • For any observed ADU values < 0, no corrections should be applied.
  • The ADU_LINEAR_[AB] values were set equal to the corresponding ADU values, for ADU > SATURAT[AB], where SATURAT[AB] are given in the extension headers and were set to the maximum count value where the fractional nonlinearity correction is <= 0.03. I think we should propagate these SATURAT[AB] values downstream in the processing, replacing the values currently in the SISPI-generated headers.
  • Recall CCD 31, amp A is noisy and its nonlinearity behavior is not stable, so that the table entries here reflect a particular realization of the linearity curve.
  • The damaged, low-fullwell CCD 61 has been treated here the same as any other CCD.

-- Huan Lin

Analysis of 20130419 PTC sequence, 23 Apr 2013

Taken with same script as 20130108 r-band PTC, including repeated 2-sec reference exposures.

Example plots for CCDs 15 and 46 are as follows, showing cleaner nonlinearity results after correction for illumination drift (though note drift is not as cleanly linear as for the 20130108 PTC): Comparisons of nonlinearity trends for example CCDs 15 and 46 for this PTC vs. the earlier 20130108 PTC show excellent agreement:

Likewise, comparisons with the 20130108 PTC of the average and standard deviation of fractional nonlinearity of all amps (except those on CCDs 31 and 61) also show excellent agreement: ptc_20130108_20130429_linearity_all_ccds.png

Additional plots, which may be seen to be very similar to the same plots for the earlier 20130108 PTC (see its section further down this page):

Tar file of data: ratioflats_ccd_20130419.tar.
Columns in files are as documented below in section for the 20130108 PTC.

-- Huan Lin

Comparison of PTC sequences, 23 Jan 2013

Comparison of 20130108 r-band PTC vs. 20120118 u-band PTC for CCD 46 (left amp): ptc_comparison_ccd_46.png.
  • Recall u-band PTC has only about half the count rate of the r-band PTC.
  • The top panel shows good agreement of fractional nonlinearity when plotted vs. median ADU.
  • The bottom panel shows disagreement of fractional nonlinearity when plotted vs. exposure time, suggesting nonlinearity trend is likely not consistent with shutter and/or exposure timing issues.

-- Huan Lin

Analysis of 20130118 PTC sequence, 23 Jan 2013

Similar to 20130111 u-band PTC, but with additional longer exposure times extending to 70 sec.

Tar file of data: ratioflats_ccd_20130118.tar.

Note exposure 170203 has been excluded from the above, as EXPTIME in the header is incorrect (value is 16.0 sec but measured counts are consistent with the requested exposure time EXPREQ = 2.0 sec.)

-- Huan Lin

Analysis of 20130111 PTC sequence, 16 Jan 2013

Same exposure times as 20130108 PTC, but using u filter instead of r filter, so that count rates are lower by about a factor of 2.

Note that because of the lower count rate, the following plots have their x-axes extended to lower ADUs cf. the corresponding plots for the 20130108 PTC:

Tar file of data: ratioflats_ccd_20130111.tar.

-- Huan Lin

Analysis of 20130108 PTC sequence, 10 Jan 2013

I've now started to analyze the 20130108 PTC sequence, which is largely similar to the 20121211 sequence, except that the exposure times have now been scrambled (by Douglas Tucker). Plots for CCDs 15 and 46 are as follows: Some points to note:
  • The data are noticeably less "glitchy" cf. the 20121211 sequence, in particular there don't appear to be any discontinuous changes in the reference exposure trend, just a linear drift. The removal of the linear drift dramatically cleans up the appearance of the data between the top panels and the bottom panels.
  • The linearity trends seen are similar to (and cleaner than) previous PTC sequences, in particular the trend above about 3000 ADU.
  • The 0.05 sec exposures still give ~85 ADU, and thus do not look like bias frames.
The following figures then illustrate the nonlinearity behavior the CCDs: Tar file with 62 data files, one for each CCD, is ratioflats_ccd_20130108.tar. The CCD is given in the file name, and the columns in each file are:
  1. Exposure ID number
  2. Exposure time (seconds)
  3. Drift correction factor (divide by this to get rid of linear drift in illumination level w/ time)
  4. Median ADU of left amp ([1:1024,*])
  5. Corrected count rate (left) = median ADU (left) / (exposure time) / (drift correction factor)
  6. Fiducial count rate (left) = median count rate (before drift correction) of reference (2 sec) exposures
  7. Fractional nonlinearity (left) = corrected count rate / fiducial count rate - 1
  8. Median ADU of right amp ([1025:2048,*])
  9. Corrected count rate (right) = median ADU (right) / (exposure time) / (drift correction factor)
  10. Fiducial count rate (right) = median count rate (before drift correction) of reference (2 sec) exposures
  11. Fractional nonlinearity (right) = corrected count rate / fiducial count rate - 1

-- Huan Lin

Analysis of 20121211 PTC sequence, 9 Jan 2013

I've now looked at the 20121211 PTC sequence, which intentionally has greater dynamic range in exposure times (0.1-22 sec) and ADU (200-40000 ADU) cf. the previous PTC data. The reference exposure is 2 sec. Plots in the same format as before for the example (low light level) "bad" and "good" CCDs are as follows:

Note the data are noisier in this sequence, given the scatter seen for the reference exposures, but the nonlinearity trends are similar to those seen in previous PTC sequences.

Then the following figure overplots all the CCDs, except CCDs 31 and 61

Here we see quite similar nonlinear trends for all CCDs. Thus a global problem with the shutter timing or the measurement of the exposure time might be responsible, though it seems strange why the nonlinear trends (neglecting the CCDs with low light level problems) are stronger at larger exposure times (~5-20 sec) and higher ADUs (> 10000 ADU).

One problem exposure I noticed is 160396, which has EXPREQ = 2.0 sec, but EXPTIME = 18.0 sec, which is erroneous, as the measured ADU level is consistent with the requested 2 sec exposure and not an 18 sec exposure. Could this indicate a problem with the measured exposure times?

-- Huan Lin

Nonlinearity at high count rates, 8 Jan 2013

Along the same lines as what Paul noted below for the A/B offset at high count rates, and similar to the analysis of the lower light level PTC data described below in my entry for 30 Oct, I've looked at some of the high light level PTC sequence from 20121024 (exposure times of 2-23 sec, with repeated fiducial exposures at 20 sec), which cover the range of about 3000-30000 ADU. The fiducial exposures show a drift in time of the illumination level of about 1%, which is again taken out using a linear fit, though there are some exposures that are noticeably offset from the best-fit linear trend.

Notice that both the (low level) "bad" CCD 15 and "good" CCD 46 show similar high light level nonlinear behavior, about 1.5-2% over the 3000-30000 ADU range. In particular see the bottom left and bottom right panels in each plot. I would suggest that if we can consistently measure the same trends with additional PTC data (including repeated fiducial exposures), then we may want to apply a linearity correction step to the processing since these trends are similar to our 2% photometry requirements.

-- Huan Lin

A/B offsets at high count rates, 7 Jan 2013

During the Fringing analysis, I noticed that many of the fringe correction frames exhibited offsets of several DN between the A and B amplifiers. These offsets do not appear consistent with purely low-level nonlinearity, as all of the input frames have a high sky background, yet it does appear to largely affect the same CCDs (e.g. 15, 31). The Y-band (z-band) fringe frames were constructed from 50s (200s) exposures, which have about 1000 (4000) DN per pixel in the sky. The offsets on the fringe frames are at the level of 5-10 DN. These offsets correspond to a fraction of a percent and are thus approximately comparable to the amplitude of the fringes (by coincidence). Here is a plot of the amplifier difference (in percent) vs. chip number for all of the z-band and Y-band fringe correction frames:

Amplifier Offsets on Fringe Correction Frames

These offsets are unlikely to impact object photometry, except for objects on the amplifier boundaries, but will likely increase noise in the coadds for objects near an amplifier boundary in any of the input frames. I suggest a constant offset be calculated and applied to one of the amplifiers to remove this offset as part of the firstcut processing.

-- Paul Martini

Repeat of stellar linearity check with IDL, 03 Dec 2012

For CCD 15 I've repeated the stellar linearity check for one of the g-band 0.1 sec images (148046) vs. the 10-sec reference image (148100), but using IDL to measure counts instead of SExtractor. This is just to verify that the SExtractor results are not affected by some issue due to the detection thresholds at low count levels, etc. The IDL results were done by first subtracting off the median ADU level independently in each of the left and right amps, and then simply summing up the remaining counts in a 11x11 pixel box (approximately twice the FWHM) centered on the SExtractor-determined centroid. The resulting plot is here:

The IDL-determined counts (green and blue points) are in good agreement with SExtractor values (red and black points).

Note this is just for one image so there are fewer points vs. before. I've also cleaned out the objects that were saturated on the 10-sec image; this was not done in the previous plots. Finally, there was a bug in the previous plot where the x-axis was really ADU/(FWHM*FWHM)/2 instead of ADU/(FWHM*FWHM) as stated in the plot.

Stellar linearity checks using BD+17 data, 12 Nov 2012

I've now done stellar linearity measurements for the g-band observations of 20121106 in the BD+17 field, where exposure times ranged from 0.1 to 10 sec. This is similar to the earlier CDFS analysis, but the measurements are now done more systematically using SExtractor, instead of manually using IRAF imexam. Here are the relevant processing and analysis details:
  • The g-band exposures used were 0.1 sec (148046,148052,148058,148064,148070), 0.3 sec (148076,148082), 1 sec (148088), 3 sec (148094), and 10 sec (148100).
  • v7 of the standalone pipeline was used, including flatfielding by the dome flats from the same night.
  • The images showed positional drifts with respect to each other, but I could not get Scamp to give good astrometric solutions, probably due to lack of stars in these short exposures. Instead, the positional registration was done using separate IDL code, and done independently for each of the 3 CCDs analyzed.
  • Objects used in the analysis have to be in the SExtractor catalog (run for Scamp) for the 10-sec exposure, and then the objects also have to have a match in the SExtractor catalogs for every one of the 9 other shorter exposures.
  • Processed images and catalogs are at Fermilab in /data/des09.b/data/hlin/standalone_pipeline_tests/v7/runindir/runfirstcut_bd17_20121106/g
  • The plots below show on the y-axis the fractional difference in FLUX_AUTO, referenced to the FLUX_AUTO value in the 10-sec exposure. The x-axis is FLUX_AUTO/(FWHM*FWHM), in ADU/pixel, which is my rough proxy for the typical ADU counts in the object above sky background (which is only about 15 ADU or so in the deepest 10-sec exposure).
Here are the resulting plots

As before, the right amps of CCDs 15 and 31 show significant nonlinear behavior at low ADU counts, in particular compared to the left amps. The two amps of CCD 46 both behave better, though they also both show deviations at the lowest ADUs; I'm not sure whether the latter deviations could be due to SExtractor photometry systematics at low count rates instead of due to nonlinear behavior. Also, the points with large deviations at high ADU counts are saturated stars which were not culled from the analysis.

Huan Lin, 12 Nov 2012

Juan Estrada's response to the droop<e> model.

I agree with your droop(e) model.
The only thing that i would add is that i believe droop(e)<>0 for both amps, because that is what the flat fields tell us (see my fits).

I also believe that droop(e) is really droop(<e>), function of the average level on the line. I think i can show you this with scope traces. Need a bit if time to send you some traces.

Finally i have to say that certainly we never looked at the linearity of these ccds with levels of 250e- (100ADU) during our testing at FNAL.


Possible interpretation of Huan's stellar data (Gary 31 Oct)

I'll explore the possibility that all of the data on the bad amp of chip 15 are explained by a single pixel-by-pixel nonlinear response, which it's useful to write as

ADU = g * e + pedestal - droop(e)

where ADU is the detector readout, e is the number of photoelectrons, pedestal is the value which is removed by overscan subtraction and hence I will further ignore, g is the gain which can be determined from the high-count regime, and droop(e) is the number of "missing" ADU in the readout due to nonlinearity.

First question is: what do the flat-fields tell us about the droop(e) function?
  • First, we know that droop(0) = 0 because the zero frames show no signal after overscan subtraction.
  • Second, Juan's measurements of good linearity with fixed A/B ratio after applying offset suggest that droop(e) reaches a constant asymptotic value of 5-7 ADU by an illumination level of about 300 ADU.
  • Third, how does droop rise from 0 to this 5-7 ADU? I took the mean sky values for the stars on the A and B amps in Huan's stars_15.dat as a measure of sky brightness. I assumed that the left-hand side is perfectly linear and serves to measure e correctly. I then calculate droop = (sky on A) * (330/307) - (sky on B) where B is messed-up side. The result is
     sky  droop  *deficit   *peak 
    2.95 1.57    1793.      32.8
    5.88 2.79    1929.      63.7
    14.9 4.86    1070.     166.7
    29.5 6.04    328.       347.6
    59.4 6.78    489.       646.5
    149. 7.85    -1223    1503.5
    307. 7.89    0            2391.5

I have chosen the value 330 / 307 as the one that makes the droop reach an asymptote at the longest exposure, with 300 ADU of sky. [We could measure this better with brighter sky or flats and remove this fudge factor in my model, which is really g(B)/g(A)]. If the model is correct, we see that the droop rises from 0 to ~8 ADU over an illumination range of 0 to ~100 ADU.

Next I summed the background-subtracted fluxes of 4 stars in each half of the array in Huan's table. I then calculate how many ADU are "missing" from the B-side stars relative to the value expected if they were strictly proportional to the A-side stars' fluxes: deficit = (total A flux) * (617227/718591) - (total B flux). The factor in parentheses is just the ratio of the stellar fluxes in the longest exposure, hence deficit=0 at the longest exposure, by construction. What we see is that the stellar flux deficit on the bad amp decreases as the exposure gets deeper. This is qualitatively as expected in the simplest nonlinearity model: on the shortest exposures, the sky is only 3 ADU and sky pixels are missing 1.6 ADU from droop. The star pixels have a bigger flux and hence bigger droop, and hence the aperture mag is too low. In the longer exposures, the sky is brighter and more of the 8 ADU total droop is already present in the sky annulus, hence the stellar pixels are not as strongly deficient in ADU's relative to the sky.

Quantitatively this is crude as there are fudge factors and noise here. But It's maybe not crazy: on the shortest exposures, there are about 1800 ADU missing from the 4 stars, or 450 ADU per star. About 2 ADU of droop are present in the sky, so highly illuminated pixels should be up to 6 ADU "droopier" than the sky. The stars have 5 pix FWHM, so about 80 pixels are within 1 FWHM of the stellar peak. If they're all 6 ADU short, we have our stellar deficit. Actually the deficit is on the large side for this simple model, but it seems worth doing this better.

We could do a better job by first mapping the droop function more carefully, then "de-drooping" or re-linearizing the exposure data on the bad amp pixel by pixel, then seeing if the stellar photometry obtains a fixed ratio between A and B amps.

In any case it appears that there is a non-linearity that becomes increasingly severe below 100 ADU, and that for now we cannot trust photometry done with low sky levels unless the star counts are so high that 5-10 ADU per pixel is unimportant.

Stellar linearity checks using CDFS data, Oct 31

Here are plots showing the trends in CDFS stellar fluxes vs. those in the PTC flats
  • CCD 15: plot3ratios_ccd_15.png
    - Top left: Comparison of fractional differences in PTC count rates (2 PTC sequences from 20121025 and 20121024) and in the CDFS sky count rate, plotted vs. median ADU level in the images. These show a strong trend for the right amp, going down to low ADU levels, as seen in the CDFS sky data. (Would be nice if there were more overlap among the 3 data sets!)
    - Top right: Now I've added CDFS stellar count rates, measured using IRAF imexam for 6 stars from the left amp (black lines) and 4 stars from the (problematic) left amp (orange lines). The fractional difference in the stellar count rates is relative to that in the longest CDFS exposure of 10 sec. For the x-axis, I've used (sky background + peak count in star/2) to represent the typical ADU level for the stars. We see that the stars appear to exhibit significant nonlinearity at low ADU levels for the right amp, consistent with the trend seen in the CDFS sky data.
    - Bottom left and bottom right: Just plotting the left and right amp stars separately for clarity.
  • Stellar fluxes measured by IRAF are available in this file stars_15.dat
  • If anyone else wants to have a look, the processed data used for the star measurements are here
    It's just the subset of 7 exposures (0.1, 0.2, 0.5, 1.0, 2.0, 5.0, 10.0 sec) I've processed so far (there are lots more, with exposures 0.1-10.0 sec, in 0.1-sec steps, in the full set of raw data in /blue-orch/DTS/src/20121028). These are overscan and master bias (from the pre-sequence biases) subtracted, but intentionally with no flatfielding.

Comments on Huan's analysis below by Gary -

My question is whether this nonlinearity can be described as a nonlinearity on every pixel, such that ADU = f(photons) where f is not quite linear; or whether this is some kind of trapping or offset effect where ADU = gain*photons - offset(background level). The latter would be in some sense benign since it is just a shift in the sky level, but if there is some kind of nonlinearity or trapping that varies pixel to pixel, that would damage the photometry much more seriously.

David James took some data (143743-143845) with exposures of a fixed field at varying exposure times. This would be amenable to this test:
  • On a given CCD, on longest exposure, select stars in different ranges of flux on both A and B amps.
  • On each exposure, take aperture photometry of these collections of stars and establish an A and B mean in each flux bin.
  • Determine A/B ratio of the stellar fluxes vs exposure time and stellar flux.

It will be necessary to sum the flux of many stars to get <1% errors on fluxes of stars with O(100 ADU) counts per star. Taking the A/B ratio will cancel out any effects of seeing variation on the aperture photometry.

I cannot even get into NCSA right now to retrieve these images, never mind reduce them. So if anyone can do this...

30 Oct

I looked at linearity for CCDs 46 (good), 15 (bad), and 31 (ugly) using the PTC sequence from 20121025 (exposure time = 0.2-2.0 sec r-band dome flats in the exposure number range 142529-142600; subsequent images in the same sequence with exposure times 2.1-3.0 sec still need to be analyzed). These data go down to low light levels (~300 ADU), and importantly also include repeated measurements at the fiducial exposure time of 2.0 sec. The fiducial data show a drift in time of the illumination level of about 0.7%, which is small but not negligible compared to the linearity trends being measured.

The PTC data were processed using v7 of the standalone pipeline, including overscan subtraction and subtraction of the master bias constructed from the 5 pre-sequence biases (exposures 142524-142528). The processed data used for the analysis below are at Fermilab in /data/des10.a/data/hlin/standalone_pipeline_tests/v7/runindir/runfinalcut_ptc_20121025/02_imcor/data

I then used IDL to calculate the median counts on each amp of each CCD (using all the data without any rejection), then divided by the exposure time, and finally plotted the resulting count rate, relative to the fiducial count rate derived from the the 2.0-sec exposures. The top panels of the following plots show results vs. exposure time (left), vs. elapsed time since the first exposure in the sequence (middle), and vs. ADU level (right). The points are color-coded by left (black) and right (magenta) amplifier, and the fiducial 2.0-sec exposure data are also distinguished by color (blue and green). In particular in the middle panel we can see the drift in time of the illumination level, as seen in the 2.0-sec exposure data. This drift is then fit to a linear function in the elapsed time, and then this fit (orange) is divided out of the data to remove the drift. The results are then plotted in the bottom panels.

  • CCD 46 (good): linearity_ccd_46.png
    - The left and right amps behave very similarly, consistent with Jim's earlier observation of a stable A/B ratio.
    - However, it does look like there is a small nonlinearity trend, amounting to ~0.5% at ~300 ADU; will be good to get even lower-ADU data to check the continuation of the trend.
  • CCD 15 (bad): linearity_ccd_15.png
    - Here the left and right amps behave differently, with the right amp off by ~2.5% at ~300 ADU, while the left amp is only off by ~0.5% at the same place (similar to CCD 46). These results are similar to what Juan measured earlier for the same data set.
    - The different linearity behavior of the two amps results in the trends seen earlier in the A/B ratio.
  • CCD 31 (ugly): linearity_ccd_31.png
    - The behavior of the right amp (magenta) is just a mess.
    - The left amp actually looks good, and seems to show the most linear behavior of all the amps examined so far, once the overall LED output drift is taken out.
I am working on adding the 2.1-3.0 sec data from the same PTC sequence, and also on extending this to the remaining CCDs. I think it will be good:
  • for someone else (Juan did for CCD 15) to independently verify these results
  • that we take additional PTC sequences
    - to check repeatability of the (non)linearity trends
    - to go down to even lower light levels (say 50 ADU, as Gary had suggested) since 100 ADU = 450 e- is the expected dark time g-band sky level in a 100-sec DES main survey exposure, and is really not a low light level for on-sky data
    - with repeated measurements at a fiducial exposure to take out any illumination level trends
  • that we take stellar linearity exposures, similar to the PTC sequence (including repeated fiducial exposures), to test this on-sky

Also, here are plots of the ratios of the row-by-row medians of the 2.0-sec to the 0.2-sec exposures, for each of the above 3 CCDs:

Except for the right amp of CCD 31, the flatfield image ratios (expressed as fractional differences) do not show any obvious spatial structure.


From Jim, 30 Oct

Juan Estrada described a model for the effect that is the first
one I've heard that seems physically motivated.

Imagine a row being read out; the serial register is shifting rows
to the amplifier which sends a voltage to the preamp, which
sends a different voltage to the A/D converter for dual correlated
sampling conversion to a digital number. This dual correlated
sampling essentially a mechanism where the zero level voltage is read,
then the pixel voltage is read, then the zero level voltage is read again.
The zero levels are averaged or interpolated and subtracted from the
pixel voltage, and the result is fed to the A/D converter and the DN measured.

Fine- now imagine that zero level wanders between the two samples of it for a single
pixel, and the amount of change is proportional to the time the serial register
had been shifting. That gives a non-linear signal as a function of column number.
The amount of change of the zero-level across the pixel read could well be
proportional to the voltage it is being given (i.e., to the number of photons) and thus
to the signal level, giving a non-linear signal as a function of photon number.
In this model one imagines the zero-level voltage is being dragged a bit
via a ground or some other coupling to the needs of the circuit to provide a voltage
somewhere. At high signal levels this drag saturates to some value, and the
device is linear.

This model shows the limitations of the usual loose way astronomers separate
completely linearity and bias as we define bias to be what is measured in
overscan or master bias frames. The changing zero-level is a changing bias,
but one that can't be measured in a bias image; it induces a non-linearity exactly
by adding a constant to a linear measurement; that constant need not be constant
with column number.

From Jim, 27 Oct

This is half a status report to those on the SV list that
haven't seen the 52 emails on the topic, and half a report
that the u-band flats show the effect even at ADU > 10,000.


There is a mystery bias in 18 of 62 ccds:
15, 18, 22, 31, 45, 51, 56, 2, 3, 12, 21, 32, 34, 41, 42, 61, 37, 44

For the mystery bias, a typical good ccd is 46,
the archtype bad is ccd 15, and the worst behaved is ccd 31.

We can clearly show the effect in a photon transfer curve data
set. While the variance in a pair of subtracted images of amp A
as a function of illumination is linear (see ptc_S16_channel_25_left.pdf,
a linearity curve from Jiangang; note the low light levels involved).

On the other hand, if we ratio the flux in a ccd's amp A to amp B,
using the same data set, we find a "non-linearity" at low counts.
(see ccd-15-3days.pdf, which shows data taken on three different
days, with the blue on the left being the same data set that produces
Jiangang's plot).

This effect is consistent with an extra 7 ADU of bias for this particular CCD
ratio = r0 * flux / (flux + offset) for offset = 7DN.
(See ccd-15-dep-ptc.pdf). One then can worry about the details of
bias subtraction, but I don't think this is an issue. In that figure
the red points are the Oct 24 PTC data set corrected using the
overscan column regions (so assuming there is no spatial structure
to the bias), and the blue points are thee PTC data set corrected
using a master bias made of 10 biases taken at the beginning and
end of the PTC sequence (so assuming there is no temporal change
of bias, as I didn't norm the master bias to individual frames). Having made
two questionable assumptions, the data might have been expected to show
the effects of data maltreatment. In fact, the data are indistinguishable for
CCD 15, though CCD-13 shows worse noise for the master bias subtraction.

So the data is linear, but there is a mystery bias in 18 ccds
at low light levels. Where does this 7 ADU come from? It isn't in the
non-illuminated pixels.

On the other hand, perhaps it isn't a big deal. The only way to get
< 1000 counts in an r-band flat is to either to expose for < 300 mlli-seconds
or by taking flats with the flat LEDs off. Clearly one of the tasks of the
observers is to make sure the flats have > 10,000 DN.

In the data, this extra bias would subtract in local sky subtraction
just as sky would.

The flats, if they show this effect, are corrupted. The easy fix is to
keep the flats at > 10,000 ADU.

That's the status.


Now comes the new part:

I looked at 690 flat fields taken during October. The A/B ratio
as a function of day is shown in ccd-31-october.pdf. There are
excusions away from the mean almost every day flat fields
are taken: the excusions downward are the photon transfer
curves, and there need not be excursions upwards associated
with these. On the other hand, normal days always seem to have them.

Looking at ccd-31-october-counts.pdf, the same data plotted
against counts, we see that the excursions upwards are for
flat fields with counts > 5000 and > 10000 counts.

These turn out to be u-band flats taken during the normal
flat sequence.

image & mjd & mjd & filter & time \\
142655 - 142661 & MJD 56226 & u & 15 secs \\
142323 - 142329 & MJD 56225 & u & 15 secs \\
141098 - 141104 & MJD 56222 & u & 7 secs \\
141009 - 141015 & MJD 56222 & u & 15 secs \\
140767 - 140773 & MJD 56221 & u & 15 secs \\
140333 - 140339 & MJD 56220 & u & 15 secs \\

Chip 31 has 2% variations in the u-band flats. Chip 15 has 1%

It'd be nice to get to the bottom of this, of course.


27 Oct 2012

Status right now is that Jim reports the effect does not change with flats taken 26 October after ion pump has been turned off.
  • Jim reports u-band flats show a different A/B ratio than all others filters, at counts > 5000, even at > 10,000 ADU
    • 142655 - 142661 MJD 56226 15 secs
    • 142323 - 142329 MJD 56225 15 secs
    • 141098 - 141104 MJD 56222 7 secs
    • 141009 - 141015 MJD 56222 15 secs
    • 140767 - 140773 MJD 56221 15 secs
    • 140333 - 140339 MJD 56220 15 secs

24 Oct 2012

See reports from Jim Annis on analysis of recent dome sequence at varying illumination in the observing log and report from Keith Bechtol at docdb-6698

  • Jim's analysis shows droops in A/B ratio (or rises) with decreasing flat level for multiple chips. Dependence is smooth in all cases except chip 31, which is worst and also time-variable.
    • The clearly badly behaved ccds: 15, 18, 22, 31, 45, 51, 56, _(these are CCDNUMs)
    • The mildly misbehaving ccds: 2, 3, 12, 21, 32, 34, 41, 42, 61
    • If I'm picky: 37, 44
  • Jim's results are robust to whether overscan is subtracted line-by-line or whether a master bias frame is just subtracted instead of overscan subtraction.
  • Gary notes that Jim's data are consistent with A/B ratios that behave as ratio = r0 * flux / (flux + offset) with offset between +-5 ADU for all CCDs except #31. #31 has noisier fit about value of offset ~ 20 ADU.
  • Gary also notes that it's the right side (higher x pixel index) that is varying with time even within 1 readout of chip #31. And overscan value is variable for this amp.
  • Jiangang shows that A/B ratio was varying for chip 31 in flats taken in early October.