Comparing Photometry to Homogenized Coadds


As a follow up to Comparing to Homogenized Coadds, let's look at the photometry. Please see that page for information on how the "bad stars" are defined.

The data

I'm starting with the field of SPT-CLJ0539-6013 , which overlaps with DES tiles DES0538-5957 and DES0544-5957.

The three catalogs compared are "Munich" from Shantanu and Joe, run with homogenized psfs (but predating GCM, plus SLR to SDSS mags); SVA1 (Gold 1.0) (GCM + including SLR to DES mags); and "lmfit", which is Erin Sheldon's (experimental) multi-epoch photometry fitting. The lmfit catalogs (also run with GCM) do not need homogenization since the PSF is well known for each individual image. I'm concentrating on z-band here because that shows the worst of the problems (but the color tests at the bottom are i-z).

Star Comparisons

We know that the psf size (and spread_model) is biased for the bad stars/bad regions. What about the stellar photometry?

MAG_AUTO (Munich, SVA1)

First, we can look at MAG_AUTO_Z which doesn't depend on the psf.

sva1 mag_auto_z - munich mag_auto_z vs sva1 mag_auto_z. Red squares are "bad stars"

It's clear that the mag_auto_z values agree (except at the brightest end near saturation where something else is going on) although there is significant scatter (3%). The bad stars are not biased in any way.

MAG_PSF (Munich, SVA1)

Next, we can look at MAG_PSF. Well, not quite. I don't have MAG_PSF for the Munich coadds (not used?) but I do have MAG_MODEL which generally converges on MAG_PSF for stars.

Yikes! We can see that while in general the photometry agrees (with relatively large 4.6% scatter) the bad stars are significantly offset (~0.1 mag). So not only is the SPREAD_MODEL biased but the stellar photometry is biased. This is something to worry about.

A digression on MAG_AUTO and MAG_PSF

As an aside, there is a significant slope in the MAG_AUTO vs MAG_PSF relation. This is generic and is seen in all images, including single epoch. I am plotting SVA1 for reference.

Something is wrong with either MAG_AUTO or MAG_PSF. My guess is MAG_AUTO, but I can't obviously guarantee that. In any event, caution when choosing your star magnitudes!


How does MAG_PSF look when performed with the multi-epoch fitting?

On the left, you can see that the Munich and lmfit agree, and the bad stars are not outliers when using the proper multi-epoch psf fitting. However, there is still significant (3.5%) scatter. On the right, the sva1 psf vs lmfit shows that the bad stars are significant outliers, as above. But, surprisingly, the scatter between SVA1 and LMFIT psf magnitudes is smaller than the homogenized coadds: only 2.8%, and fewer outliers. I find this surprising, but it may simply be caused by using MAG_MODEL which has more parameters than MAG_PSF.

Star Conclusion

  1. MAG_AUTO is biased relative to MAG_PSF
  2. The stellar photometry for the bad stars in the bad regions is biased in SVA1
  3. The stellar photometry for the bad stars in the bad regions is unbiased with LMFIT
  4. There is a surprising amount of scatter between LMFIT psf mags and Munich model mags. Maybe this is due to the use of MAG_MODEL for stars.


So the stars are biased, but these are much more sensitive to getting the exact shape of the psf correct. What about the galaxy photometry? Galaxies were selected as i>22 and SPREAD_MODEL_Z > 0.002 in the homogenized coadd catalog.

Bad Regions

First, I define a couple bad regions based on the locations of the bad stars. (I don't have a similar clean selection of bad galaxies from the psf fitting):

Bad stars in red, bad galaxies in "bad regions" in blue.

MAG_AUTO (SVA1, Munich)

First, let's look at MAG_AUTO comparisons. Red points are galaxies in bad region:

There is significant (4.5% at i<20) scatter but the galaxies are unbiased which isn't surprising since MAG_AUTO doesn't use the psf model (and similar to the stars above).

MAG_MODEL (SVA1, Munich, lmfit)

Next, we look at the MAG_MODEL comparisons for sva1 and Munich. Red points are galaxies in bad region:

The first thing to note is that there is a ton of scatter: 12% for i<20. Presumably, at least part of the scatter is due to the different models assumed. The SVA1 run was done with a disk-only (aka exponential aka sersic n=1) model, while the Munich reductions were done with a different model (I'm not sure of the profile, but I'm fairly sure it wasn't disk only). Nevertheless, the scatter is surprising. Meanwhile, it looks like the galaxies in the bad region are unbiased, which is good.

If we compare MAG_MODEL in SVA1 to the lmfit (also using disk only) things are much better behaved:

The scatter is much lower (4%), presumably dominated by the better psf fits in the multiepoch fitting. (I also note that there is a significant bias at the bright end when comparing the lmfit n=4 magnitudes to the sva1 n=1 model mags. The perils of using the wrong galaxy model, at least at the bright end!)

Again, the galaxies in the bad region are unbiased. (The zeropoint offset is partly due to the SLR applied to the SVA1 catalog, and partly due to an as-yet-undetermined zeropoint offset between the SVA1 coadds and the lmfit values.)

MAG_DETMODEL colors (SVA1, Munich, lmfit)

The galaxy colors are where the rubber hits the road. I'll concentrate on i-z here.

First, MAG_DETMODEL colors for sva1 and Munich (on the left) and sva1 and lmfit "match" colors on the right. These are derived similarly to detmodel (simultaneous fit to multiple bands), with the addition that the color is based on the best fit of the n=1 or n=4 sersic model. (An aside: the following plot looks the same using pure exponential/n=1 colors):

Left: SVA1 i-z detmodel color - munich i-z detmodel color Right: SVA1 i-z detmodel color - lmfit i-z match color. Note different scale.

Whoa, on the left panel there is a lot of scatter here (10%), which is very worrying. Plus there's a bias at the faint end. Plus the bad region galaxies also appear to be biased relative to the main locus, however, due to the shape it's hard to tell how significant the bias is for the galaxies in the bad region.

The scatter is much, much smaller on the right panel (note the different scale!), only 2%. (The offset is due to the lack of SLR corrections on the lmfit colors). The galaxies in the bad region appear to be biased in color by ~1% (in the median) which is a problem for precision photometry but not nearly as bad as the star psf mag biases. It seems that the extended sources make the photometry less susceptible to biases caused by slight mismatches in the psf.

Galaxy Conclusion

  1. The photometry biases caused by the wrong psf in the bad regions appears to be much smaller for galaxies than for stars. This shows up as ~1% biases in color.
  2. MAG_AUTO is not affected by the bad regions (as expected)
  3. There is large scatter (12%) in MAG_MODEL between the SVA1 and Munich reductions, perhaps caused by the different models.
  4. There is also large scatter (10%) in the MAG_DETMODEL colors between SVA1 and Munich. These colors should be less susceptible to differences in models because the offsets tend to cancel. (For example, the n=4 colors using lmfit look almost exactly like the n=1 colors even though the individual magnitudes are biased. This has also been seen with other DETMODEL tests.)
  5. Because lmfit colors which use the full psf information in each individual epoch agree with the sva1 colors (with 2% scatter) then it seems that the sva1 colors in this field are more reliable than the Munich reductions for the galaxies.

Another Field, Comparing to Finalcut

As an alternative, let's look at another field (SPT-CLJ0458-5741, overlapping DES0457-5748). And instead of the multiepoch fitting from lmfit, let's look at a (weighted) average of all the measurements from SVA1 finalcut with the GCM zeropoints applied. This field was chosen basically out of a hat; I'm not sure if the Munich reductions have the pixel offset bug or not.

All comparisons are for MAG_AUTO, because we don't have MAG_PSF for the Munich catalogs and we don't have MAG_MODEL or DETMODEL for the finalcut catalogs. But MAG_AUTO appears to be showing some of the same problems as MAG_DETMODEL above.


In all plots the bias and scatter are given in the corner. I'm not too worried about the bias at the moment, what with SLR corrections and all.

There is more scatter in the Munich reductions relative to finalcut (which of course has a well understood psf, but this is not using psf fitting). Are there key starflat and other illumination corrections that are important? Or GCM vs PSM? There are actually a lot of variables to consider when comparing the two reductions, not just the homogenization.


In all plots the bias and scatter are given in the corner. I'm not too worried about the bias at the moment, what with SLR corrections and all.

Once again, there is considerably more scatter in the Munich reductions.

A quick red-sequence test

The intrinsic width of the red sequence is 0.02 (i-z, r-i) to 0.05 (g-r) mag at z<0.4, so we can look at the photometric width of the red sequence to see if something is going wrong with one or the other catalogs. We want to stick to relatively low redshifts to ensure that we can get enough galaxies (somewhat arbitrarily cut at 0.2L*, per redmapper) that are bright enough that we are measuring the intrinsic width rather than photometric noise.

Two of the SPT clusters with data available are appropriate: SPT-CLJ2351-5452 (z=0.39, lambda = 91) and SPT-CLJ0458-5741 (z=0.19, lambda=42). The first of these is obviously better because it is richer...

I've taken all the galaxies within 1 Mpc of the center and brighter than ~0.2L* (i<21.3 for SPT-CLJ2351-5452). Galaxies are matched from one catalog to the other, so that the same galaxies are plotted. Then I simply plot the histogram ... a tighter red sequence will yield a more concentrated core. (To do the full statistics on the intrinsic width requires more clusters, more sophisticated fits, etc.)

White: Munich. Red: SVA1 Gold White: Munich. Red: SVA1 Gold

It is clear in both of these colors that the red sequence is significantly tighter in the Gold catalog than the Munich catalog, indicating that there is significantly greater photometric scatter in the Munich catalog.

For the lower redshift, poorer cluster (SPT-CLJ0458-5741), the statistics are much lower and there is no obvious difference in the histograms for any of the colors between the two catalogs. This is curious in light of the apparent increase in photometric scatter compared to the finalcut averages (above), but it may be a lack of statistics or it may be that the scatter is mostly with the non-red galaxies.

Finally, I note that the Munich reduction of SPT-CLJ2351-5452 has the pixel offset bug which may be part of the problem. However, the g-band image of SPT-CLJ0458-5741 also has the pixel offset bug and this does not show up as a problem with the red sequence so this may be a red herring.