Project

General

Profile

Testing Completeness with UFIG

Chihway Chang and the Zurich crew are making a full set of SVA1 footprint test images with realistic sky, seeing, etc. Right now it is coadd images which are simulated, and each tile has a uniform sky and seeing and psf. But over the full SVA1 we'll be able to sample a wide range of sky and seeing values which will be very useful. Galaxy and star positions come from the BCC. Each coadd is then run through sextractor using the same settings as SVA1.

Testing on UFIG SVA1 v0.4

-Eli.

The new UFIG runs are now up to version 0.4! with many improvements!

All plots here are for old v0.3 (left) and new v0.4 (right)

Completeness

Completeness is estimated from UFIG using the truth, and for the red points from the shallow re-coadds of the deep SN fields.

UFIG v0.3 UFIG v0.4

And residuals from a best-fit line

UFIG v0.3 UFIG v0.4

Very similar, and the new version is a bit more consistent than before. This is good!

And for the width of the error function in the completeness model:

UFIG v0.3 UFIG v0.4

Limiting magnitude Comparisons

How does the limiting magnitude for the 2" aperture magnitude compare to that of mag_auto and mag_model? (Note that mag_model may not match perfectly: while in both cases we are using the same disk-only model, the Sersic indices in UFIG are not ideal, so the shapes of the faint galaxies in UFIG are not necessarily the same as those in the data.)

In this case we can also add blue points for the SVA1 data. Galaxies were split into 10 bins of maglim_auto_i according to my depth maps.

UFIG v0.3 UFIG v0.4

The aper/auto comparison looks very good in v0.4.

UFIG v0.3 UFIG v0.4

This looks much the same between v0.3 and v0.4, and seems ... okay.

Magnitude Offsets

For typical galaxies at mtrue=20 and 23 (i band), what is the typical offset between aper and auto magnitudes, and model and auto magnitudes? We can do this both for UFIG and for the data (using mag_auto as mtrue ... because it's what I have.) I have plotted these as a function of limiting magnitude and as a function of seeing.

Aper-Auto Offsets as a function of limiting magnitude

UFIG v0.3 UFIG v0.4
UFIG v0.3 UFIG v0.4

v0.4 is much improved, though not perfect.

Aper-Auto Offsets as a function of seeing

UFIG v0.3 UFIG v0.4
UFIG v0.3 UFIG v0.4

Model-Auto as a function of limiting magnitude

UFIG v0.3 UFIG v0.4
UFIG v0.3 UFIG v0.4

These look very similar. But note that something very strange is going on with the model fits near the limiting magnitude. (in both data and UFIG...)

Model-Auto as a function of seeing

UFIG v0.3 UFIG v0.4
UFIG v0.3 UFIG v0.4

The bright objects, comparing model to auto as a function of seeing is the one place where v0.4 looks a bit worse than v0.3.

Testing on UFIG SVA1 v0.3

-Eli.

The new UFIG runs are now up to version 0.3!

A whole bunch of bugs have been fixed, the noise levels are better calibrated, etc. Note that the UFIG tiles don't match onto coadd tiles, but this will be changed in the next version. However, as the coadd systematics vary significantly over a tile, it's worth noting that we don't necessarily expect an exact 1-1 match in depth from coadd tiles to UFIG tiles without first decomposing the coadd tiles into different depth pixels.

Measuring the completeness

For these tests I have used only "true" galaxies, to make sure we aren't getting completeness effects confused with star/galaxy separation issues. I have not simulated a star mask, so that will decrease the completeness from 1.0 where some stars are masked out.

For each tile, I measure the MAG_AUTO limiting magnitude in r, i, and z. I also measure the completeness using an error function along the lines of Tom Diehl:

completeness = (eff/2)*(1-erf((mag - m50)/sqrt(2*w)))

Where eff is the efficiency at the bright end; m50 is magnitude at 50% completeness (for 5sigma detections), and w is the width of the turnoff. Here is the model fit for an arbitrarily chosen tile:

Left: UFIG tile 102 Right:DES0224-0458 shallow coadd in SNX3

The blue line denotes the 10sigma galaxy limiting magnitude. The x axis is labeled as "mag_auto_i", but it actually is "mag_true_i" adjusted to match the zeropoint of mag_auto_i to remove any residual biases (which are another issue...). In this case, limmag_i = 22.89 (10sigma), and m50=23.61, w=0.24, eff=0.95.

  • DES Requirement R4 is 97.5% completeness at the 10sigma limiting magnitude depth in a 1.5" aperture. (This is explicitly excluding area occulted by stars.) I have yet to look at this quantity explicitly, but we are actually very close to the target of 0.975 at 10sigma MAG_AUTO limiting magnitude. Of course, the MAG_AUTO limiting magnitude is much shallower than our target, partly because this is the wrong metric to compare to specs.

As of 5/12/14 I have not yet done the mapping as a function of magnitude and surface brightness.

Completeness As a Function of Depth

The MAG_AUTO completeness parameter m50 is now almost perfectly correlated with the MAG_AUTO 10\sigma limiting magnitude. This is somewhat surprising: in this test the only thing that matters is the i band depth, even though the detection image is r+i+z. (Similarly, m50_z is only correlated with limmag_z). There's something about using 5sigma detections via MAG_AUTO in a particular band that is erasing all other knowledge of the other bands.

Even more intriguing are the red points. These are the completeness parameters for the two SNX3 fields that I have analyzed (described below). They fall right on the correlation! Does this mean that we can simply use the UFIG calibration? Not so fast...

Completeness Compared to Balrog

Erics Huff and Suchyta have run Balrog simulated galaxies on i-band images in a region of SPTE. I have binned the sky by local depth, and computed the completeness function for these galaxies:

White is UFIG, red is Balrog

The good news is that in Balrog the m50_i parameter is also very tightly correlated with the limiting magnitude as estimated from the errors. But there is a significant offset such that Balrog estimates that the completeness limit is 0.2mag shallower for any given limiting magnitude. There is also a very large disagreement as to the width of the error function, where the completeness drops off a lot faster in Balrog than in UFIG. Not sure what to make of this.

Testing on UFIG SVA1 v0.2 (OBSOLETE)

-Eli

I have run completeness tests on UFIG SVA1 v0.2 . Along with the limitations described above, there is one significant shortcoming: due to a bug, the Sersic index drawn from a separate random distribution for each individual band. Although exactly how to simulate color gradients in UFIG is not entirely clear, galaxies will be much closer to a constant Sersic index (especially red galaxies!) than a randomly drawn Sersic index. This will definitely impact color measurements, and probably detectability in r+i+z. As of 4/22/14, Chihway is starting a new run that fixes this bug (and records the Sersic index as well). With these caveats in mind...

Measuring the completeness

For these tests I have used only "true" galaxies, to make sure we aren't getting completeness effects confused with star/galaxy separation issues. I have not simulated a star mask, so that will decrease the completeness from 1.0 where some stars are masked out.

For each tile, I measure the MAG_AUTO limiting magnitude in r, i, and z. I also measure the completeness using an error function along the lines of Tom Diehl:

completeness = (eff/2)*(1-erf((mag - m50)/sqrt(2*w)))

Where eff is the efficiency at the bright end; m50 is magnitude at 50% completeness (for 5sigma detections), and w is the width of the turnoff. Here is the model fit for an arbitrarily chosen tile:

The blue line denotes the 10sigma galaxy limiting magnitude. The x axis is labeled as "mag_auto_i", but it actually is "mag_true_i" adjusted to match the zeropoint of mag_auto_i to remove any residual biases (which are another issue...). In this case, limmag_i = 22.96 (10sigma), and m50=23.41, w=0.17, eff=0.97. DES Requirement R4 is 97.5% completeness at the 10sigma limiting magnitude depth (as I read it). (This is explicitly excluding area occulted by stars.) According to this tile in UFIG (and in general in UFIG, but see below), assuming the efficiency loss is entirely due to star occultation (not an unreasonable assumption), then we are still at a completeness < 0.975 at the 10sigma limiting magnitude. This is an issue!

An aside: which galaxies are being missed? They tend to be lower surface brightness galaxies. However, I cannot quantify this until the next run of UFIG with the output of the Sersic indices (And the correct Sersic indices!)

Completeness As a Function of Depth

After running through all the tiles, we can see how the completeness parameters vary with tile (and systematics, depth, etc). The efficiency is very stable: 0.97+/-0.007 over 400 tiles; similar with w = 0.16+/-0.02. However, the m50 parameter varies with nominal i-band depth (left). (This is equivalent to the mangle depth for a 2" aperture). Things are tighter with the average of the r,i,z depth values (relevant, as the detection image is r+i+z).

However, there are still some outliers. These can be taken into account by using the measured MAG_AUTO limiting magnitude (which can also be reconstructed from the systematics maps + mangle maps):

This is very promising, that we can apparently recover the m50 value from the combined depth of the detection image over a wide range of conditions. Note also the deep SN-X3 fields in the upper right corner. These are a little shallower in completeness than an extrapolation, but it's not too far off. See below for how these compare to the "real" data.

Completeness Using the SNX3 Field

As an alternative to UFIG, we can also look at the SN fields. It turns out that for SNX3 there were many 90s images taken in SV, which allows us to get a more direct analog to "full depth" (modulo dithering) as compared to the regular SN observing and COSMOS fields which have observations that are longer than the nominal wide-field survey. The number of 90s exposures in griz are {14,16,16,13} respectively. These were generally taken under worse seeing conditions than typical, but such is life.

To simulate "full depth" I have coadded random sets of 10 90s X3 images in each of griz using the same parameters as used in SVA1 (and SVA1 GCM zeropoints). All executables were from eups "setup METAPACK floating+9". This includes swarp 2.36.2+1, which has fixed the FSCALASTRO VARIABLE bug from the version of swarp used in the actual SVA1 processing. I will soon write up details on these coadds and make them available.

After applying the same SVA1 Gold 1.0 mask to the deep X3 SVA1 coadds and the full-depth X3 coadds I measure the completeness for two fields in the center of X3: DES0224-0458, DES0227-0458.

One question to ask is whether the X3 deep coadd is deep enough to be considered "truth". Very good question! At least according to the UFIG tests above, they are. That is, m50 for the deep SN tiles is >> the wide-field survey tiles. So it does not appear that there is a population of (say) i~23.5 galaxies in the simulation that are not captured by the UFIG tests at SN tile depth.

Here is the same m50 plot as above, now with two red points for the X3 tiles:

And it turns out that the actual images appear to be significantly more complete than the UFIG tests. I guess this is a good thing. Before I dig any further, I want to rerun with the updated version of UFIG. In the meantime, we can compare the completeness for DES0224-0458 with that of a UFIG tile with matched depth in r, i, and z (As well as r+i+z of course):

Completeness for DES0224-0458 (vs deep coadd) UFIG completeness for matched tile

Note that the eff (0.98) and w (0.18) parameters in the simulated tile are in line with the UFIG distribution.

Completeness -- orig. tests (OBSOLETE)

There are many things that can be tested from the UFIG sims, including magnitude measurements, star/galaxy separation, etc etc. Right now I'm focusing on completeness since this is of primary importance to many science analyses that are ongoing. As a first test, I've used the Truth to do star/galaxy separation, so as not to confuse the two issues. Chihway has run 1 arbitrarily chosen tile through the final pipeline for testing; things look good so she's going to run through the full SVA1 starting this weekend. We will have access to the input "Truth" catalog; the output sextractor catalog for grizY; a matched catalog for all the sources; and UFIG images for grizY. As with SVA1, all detections are made on the r+i+z coadd.

The particular test field that was run was not one of the deepest tiles. Depth values are 2" aperture depths.

depth (grizY): 23.650094986,23.7795734406 ,22.9926662445,22.0859298706,20.6405487061
seeing (grizY): 1.406216781,1.327268727,1.027072008,1.070055603,1.035148437
airmass (grizY):1.497155,1.398259,1.3424841,1.462845,1.4191507

As a first test, I've looked at i-band, 5 sigma detections:

Here the white line is the completeness as a function of the true magnitude, and the vertical dashed line shows the 10\sigma limit as estimated from the magnitude error. (Note that this is significantly shallower than the nominal full depth ... or the typical sva1 depth!) The red line shows the completeness for galaxies where the observed mag agrees with the true magnitude within 4 sigma (after correcting for any magnitude bias in a given magnitude bin).

The first thing to notice is that the completeness isn't 100% at relatively bright magnitudes (=~21). This is due to the field being in a relatively dense star region (dec=~-60). Once I estimate a star mask, this should be at 100% (NACHO: On DC6B tests, with a 'standard' star occultation, the effect amounted to 2%). The second thing to notice is that the red is much below 1. This is an indication that the errors at bright magnitudes are significantly underestimated, though the faint errors appear to be correct. (I will better quantify this). Finally, we can see the rollover in the completeness brighter than the 10\sigma limiting magnitude. I believe that this does not meet the (strangely worded) requirements. A cursory inspection shows that the missing galaxies tend to be larger (lower surface brightness) than the typical galaxies at any given magnitude.