Project

General

Profile

SVA1 Gold Catalog v1.0

Contact: Eli Rykoff

The "SVA1 Gold Catalog v1.0" is now available! Currently, we have flat files with photometry and a healpix 0/1 footprint mask with nside=2048, with a total coverage of 254.4 deg^2 with griz coverage north of Dec>-61. A total of ~223.6 deg^2 of area also has Y band. These files are also ingested into the DESDM database as the SVA1_GOLD table (also in Barcelona or Brazil?). All objects are tagged with COADD_OBJECTS_ID from the DESDM database, for cross reference. I (Eli) have tried to provide what I think are the most useful columns for easy downloading, but there are others that may be missed. Please let me know!

Photo-z catalogs do exist, but they are not yet linked from this page.

Overview

The idea in the creation of this catalog was to be as generous as possible with the cuts consistent with a "science-ready" galaxy catalog. The footprint selection was made for coverage with at least 1 CCD depth in all of griz. Y was not required because this would remove the SN fields. In addition, a cut was made with Dec>-61 to remove the LMC. Although this does remove ~40 deg^2 from the SPTE region, which is a significant hit, our SLR tests show that the LMC cannot be accurately calibrated to the same scale as the rest of the survey, and thus it has been dropped. (Extragalactic science is also much more difficuly behind all the stars). This cut also has the advantage of removing 5+ deg^2 contaminated by stray light from R Doradus , the second brightest star in the infrared sky.

Additional footprint cuts were made to remove regions with (a) very high density of "crazy color" objects (due to satellites, airplanes, and stray light), and (b) very low density of galaxies (primarily at the edges and near other masked regions). In total, these cuts removed 1.5% of the remaining area.

All magnitudes are corrected to Keith's SLR v6. , and all fluxes are put on a "picomaggie" (DES AB magnitudes, ZP=30.0) system. Please see below for details.

So far, this release only contains photometric quantities and a footprint mask. Soon we will have photo-zs derived from this photometry, as well as seeing and SLR-corrected mangle masks which will directly estimate the local auto, model, detmodel, and psf limiting magnitudes.

This catalog is listed as a "substitute good" catalog and will soon be available in DESDM as a drop-in replacement for raw SVA1 photometry.

Important Notes

  1. Objects with questionable colors (g-r) < -1, (g-r) > 4, (i-z) < -1, (i-z) > 4, have not been removed from the catalog. Although these are "crazy" colors, at the faint end there are possibly drop-outs with slightly nutty colors. However, regions with a high concentration of crazy colors have been masked out as these are caused by satellites, airplanes, and scattered light. If you are not looking for drop-outs, it is strongly recommended that galaxies with crazy DETMODEL colors (as DETMODEL is less noisy in regions of varying depth) should be removed.
  2. In general, DETMODEL colors appear to be the best behaved and lowest scatter.
  3. For total magnitudes, MAG_AUTO and MAG_MODEL both have advantages and disadvantages.
  4. For stars, MAG_PSF is recommended (and these were used for the SLR corrections.)
  5. Due to the fact that the catalog was first generated before the mangle masks were available, the approximate mask used to create the footprint is about 2% smaller than the mangle footprint (mostly due to resolution issues). Therefore, there are a few holes near the edges that otherwise appear to have good imaging. Note that the good field mask knows about these holes, so the mask is correct -- it's just a bit less area than would be there if mangle were used for the original definition.

Flat File Format and Description

The data is currently available as a set of flat files. Each file has every object within the footprint. The first "basic" catalog file has RA, DEC, flags, etc. Each set of magnitudes and fluxes (AUTO, MODEL, DETMODEL, PSF, APER_3 (1.5"), APER_4 (2")) are in their own files. Use the standard DES access to the files:

Gold 1.0.4

This is a minor modification to Gold 1.0.2. The only difference is an additional list of recommended masking based on bright stars and junk objects

The files are all the same, with the addition of a badflag file, which is described here.

Badflag File

And a replacement good region mask described here.

Gold 1.0.2

UPDATED : What 1.0.1 should have been: In 1.0, OBS_Y was wrong in some cases, plus the MU values had not been corrected for zeropoint shifts. This has now been fixed! Plus, bad magnitudes are all 99.0 exactly (rather than having the zp_shift applied).

Basic Catalog File
Auto
Model
Detmodel
PSF
Aper_3 (1.5 arcsec)
Aper_4 (2.0 arcsec)
Spread model
Mu
Flux Radius
AB_image and Kron Radius
Windowed Positions
Good Field Map

Object-by-object systematics from v2.0 maps
Airmass
CCD Count
Exposure Time
PSF FWHM
Magnitude Limits
Sky Brightness
Sky Sigma

In addition, Matias Carrasco-Kind and the UIUC group has kindly made available some additions to the catalog in the form of TPZ photo-zs and SG separation values.
[[des-photoz:TPZ photo-z PDF and N(z) using sparse representation SVA1 Gold|Check this page for info and downloads]].

Some additional files from Nacho have been included in which each object's COADD_OBJECT_ID, RA and DEC has been linked to the value of v2.0 of the [[des-lss:Systematics_in_SVA1|systematics maps]] (one file per band). These include FWHM, SKYBRITE, SKYSIGMA, AIRMASS (mean and median value for all of these), ccd count, EXPTIME and mag limit.
Systematics v2.0 g band
Systematics v2.0 r band
Systematics v2.0 i band
Systematics v2.0 z band
Systematics v2.0 Y band

Gold 1.0.1 -- CORRUPTED DO NOT USE

UPDATED : In 1.0, OBS_Y was wrong in some cases, plus the MU values had not been corrected for zeropoint shifts. This has now been fixed!

Basic Catalog File
Auto
Model
Detmodel
PSF
Aper_3 (1.5 arcsec)
Aper_4 (2.0 arcsec)
Spread model
Mu
Flux Radius
Good Field Map

Gold 1.0

NOTE: Mu file has blank COADD_OBJECTS_ID

Basic Catalog File
Auto
Model
Detmodel
PSF
Aper_3 (1.5 arcsec)
Aper_4 (2.0 arcsec)
Spread model
Mu
Flux Radius
Good Field Map

Other than the footprint cuts (described in detail below), the only requirement for an object to be listed in this catalog is that FLAGS_I <=3 AND NITER_MODEL_G > 0 AND NITER_MODEL_R > 0 AND NITER_MODEL_I > 0 AND NITER_MODEL_Z > 0. (These flags guarantee that the object had non-zero weight in each of the griz images).

Basic File Fields

Basic Catalog File

  • COADD_OBJECTS_ID: id from the DESDM table
  • TILENAME: coadd tile name
  • RUN: desdm run name
  • OBJECT_NUMBER: SExtractor "NUMBER" in the flat catalog file
  • ID: an "SVA1 gold" id. May be convenient as a long integer (vs. COADD_OBJECTS_ID long64), but can be ignored.
  • RA, DEC
  • MODEST_CLASS: Modest star/galaxy classification. Galaxies are 1, stars are 2.
  • FLAGS_G/R/I/Z/Y: SExtractor flags from DESDM
  • OBS_G/R/I/Z/Y: 1 if object in footprint, 0 if out of footprint. Only really useful for Y band (all others are = 1 always). This distinguishes between Y band observations that are below the limiting magnitude, and those that are out of the footprint entirely.
  • ZP_SHIFT_G/R/I/Z/Y: SLR zeropoint shift from Keith's SLR v6. . Note that ZP_SHIFT_Y = 0 always for v1.0

The ZP_SHIFT values are stored for reference to the DESDM catalog (and can be regenerated with Keith's code and maps). They should not be used on the values in the SVA1 Gold v1.0 catalogs!

A link to the raw fits catalog can be generated by substituting the appropriate RUN and BAND:

https://desar2.cosmology.illinois.edu/DESFiles/desardata/OPS/coadd/[RUN]/coadd/[TILENAME]_[BAND]_cat.fits

and the image:
https://desar2.cosmology.illinois.edu/DESFiles/desardata/OPS/coadd/[RUN]/coadd/[TILENAME]_[BAND].fits.fz

Magnitude/Flux File Fields

Auto
Model
Detmodel
PSF
Aper_3 (1.5 arcsec)
Aper_4 (2.0 arcsec)

For example, using MAG_DETMODEL:

  • COADD_OBJECTS_ID: id from DESDM table. This matches 1-1 with the basic table
  • MAG_DETMODEL_G/R/I/Z/Y: SLR corrected DETMODEL magnitude
  • MAGERR_DETMODEL_G/R/I/Z/Y: Error on the magnitude
  • FLUX_DETMODEL_G/R/I/Z/Y: SLR corrected DETMODEL flux. Units of "picomaggies" (DES AB system, ZP=30.0).
  • FLUXERR_DETMODEL_G/R/I/Z/Y: Error on the flux. Units of picomaggies.

To convert from flux to magnitude you can simply do:

mag = 30.0 - 2.5*log10(flux)

or
mag = -2.5*log10(flux/10^12)

which is where the "pico" in "picomaggie" comes from if anybody cares.

It should be noted that the flux data contains more information since it can safely go negative.

Files are provided for MAG_AUTO, MAG_MODEL, MAG_DETMODEL, MAG_PSF, MAG_APER_3 (1.5"), MAG_APER_4 (2"). Note that there are no fluxes in the MAG_PSF file because DESDM database does not record FLUX_PSF.

Spread Model File Fields

Spread model

  • COADD_OBJECTS_ID: id from DESDM table. This matches 1-1 with the basic table
  • SPREAD_MODEL_G/R/I/Z/Y: Spread model from DESDM
  • SPREADERR_MODEL_G/R/I/Z/Y: Error on spread model from DESDM

Mu File Fields

Mu

  • COADD_OBJECTS_ID: id from DESDM table. This matches 1-1 with the basic table
  • MU_MAX_G/R/I/Z/Y: mu_max from DESDM
  • MU_EFF_MODEL_G/R/I/Z/Y: mu_eff_model from DESDM

Flux Radius File Fields

Flux Radius

  • COADD_OBJECTS_ID: id from DESDM table. This matches 1-1 with the basic table
  • FLUX_RADIUS_G/R/I/Z/Y: flux_radius from DESDM

Windowed Position File Fields

Windowed Positions

  • ALPHAWIN_J2000_G : windowed RA, g
  • DELTAWIN_J2000_G : windowed Dec, g
  • ALPHAWIN_J2000_R : windowed RA, r
  • DELTAWIN_J2000_R : windowed Dec, r
  • ALPHAWIN_J2000_I : windowed RA, i
  • DELTAWIN_J2000_I : windowed Dec, i
  • ALPHAWIN_J2000_Z : windowed RA, z
  • DELTAWIN_J2000_Z : windowed Dec, z
  • ALPHAWIN_J2000_Y : windowed RA, Y
  • DELTAWIN_J2000_Y : windowed Dec, Y

Bad Flag File Fields

Bad Flags

  • BADFLAG: bitwise flag

Flag definitions:

  • 1 (2^0): BADMASK10 (10% worst junk regions masked. Probably okay)
  • 2 (2^1): BADMASK04 (4% worst junk regions masked: Probably bad)
  • 4 (2^2): NEAR_2MASS (near bright 2MASS star: Very likely bad)
  • 8 (2^3): JUNK_DPOS (large offset in g and i band windowed positions. This object is junk.)

These have been sorted so that high flags are the worst. Recommended cuts are to remove objects with 2 or 4 or 8 flags. This can be accomplished with a bitwise AND (& in python and c) with 14, such that if ((badflag & 14) > 0) then the object is bad. You can also cheat and keep objects with badflag <= 1 which accomplishes the same thing in this particular case.

AB_image and Kron Radius File Fields

AB_image and Kron Radius
  • COADD_OBJECTS_ID: id from DESDM table. DOES NOT match exactly with the basic table, use COADD_OBJECTS_ID
  • RA, DEC
  • A_IMAGE: a_image from DESDM
  • B_IMAGE: b_image from DESDM
  • KRON_RADIUS: kron_radius from DESDM

Good Region Map

Version 1.0.4

The new Good Region Maps are based on a more conservative bright star mask as well as a bad region mask, described here

There are a few masks, with stars only; stars + 4% worst regions; stars + 10% worst regions. The 4% mask is recommended. Objects in these regions are flagged in the badflag file. Usage is the same as the 1.0.2 version.

Warning: if you download these files with Google Chrome, it will auto-gunzip but not rename the file, making it (a) very large, and (b) not be properly readable by healpy. So use wget, or firefox, or something that isn't Chrome.

Version 1.0.2

Good Field Map

This is a healpix map with nside=2048, NEST ordering scheme. The values are:

  • -1.6375e30 (healpix UNSEEN): no observations
  • 0: At least one ccd nominally within SVA1, but determined to be bad
  • 1: Good pixel!

To use this map in python, to determine if a given list of ras/decs is in a good or bad region (requires healpy and numpy):

import healpy as hp
import numpy as np

map = hp.read_map(region_file, nest=True)
nside = hp.npix2nside(map.size)
theta = (90.0 - dec)*np.pi/180.
phi = ra*np.pi/180.
pix = hp.ang2pix(nside,theta,phi,nest=True)
good, = np.where(map[pix] == 1)

And to use this map in healpixified IDL:

HIDL> read_fits_map,filename,map,nside=nside,ordering=order
HIDL> theta = (90.0 - dec)*!dpi/180.
HIDL> phi = ra*!dpi/180.
HIDL> ang2pix_nest, nside, theta, phi, pix
HIDL> good = where(map[pix] eq 1)

Pixelized Mangle Masks

Pixelized versions of the mangle masks are now available! Please see Pixelizing And Modeling Mangle Masks for details.

SVA1 Gold v1.0 Footprint Masks

The following pixelized mangle masks have been consolidated and combined with the "Good Field" mask, so can be used with SVA1 Gold directly. There are two mask files for each band. The first is the average weight sampled over 64 subpixels. The second is the fraction of those subpixels that have detections. Therefore, an additional cut should probably be made where fracdet >= 0.8 or so which will turn the sampled star holes into healpixels. See Pixelizing And Modeling Mangle Masks for details. All files are in NEST format, with equatorial coordinates.

Gold 1.0 g-band nside=4096 weights
Gold 1.0 g-band nside=4096 fracdet
Gold 1.0 r-band nside=4096 weights
Gold 1.0 r-band nside=4096 fracdet
Gold 1.0 i-band nside=4096 weights
Gold 1.0 i-band nside=4096 fracdet
Gold 1.0 z-band nside=4096 weights
Gold 1.0 z-band nside=4096 fracdet
Gold 1.0 Y-band nside=4096 weights
Gold 1.0 Y-band nside=4096 fracdet

Systematics And Magnitude-Appropriate Masks and Depth Maps

NEWER IMPROVED VERSION 1.0.2-4 (August 2014)

The new (1.0.2-3) versions include SN-S
The newest (1.0.2-4) versions are based on the latest systematics maps

Details see Improved Gold 1.0 Depth Maps

Note that there are some regions near the edges (about 2% of the total area) that are in the catalog, and are in the "good region" mask but are not in these depth maps. This is being investigated. However, these regions are really the most marginal of all and it would not be a bad idea to cut them.

The new 1.0.2-4 versions are based on on the latest v2.0.1 of Boris' systematics maps . These are the July 2014 versions, with many fixes and tweaks. Most notably, I am using the "coaddweight_mean" versions of the maps which use flux and exposure weighting. This should help the map quality in the SN regions where the exposure times are different.

MAG_AUTO

Gold 1.0.2-4 g-band nside=4096, AUTO
Gold 1.0.2-4 r-band nside=4096, AUTO
Gold 1.0.2-4 i-band nside=4096, AUTO
Gold 1.0.2-4 z-band nside=4096, AUTO
Gold 1.0.2-4 Y-band nside=4096, AUTO

MAG_DETMODEL

Gold 1.0.2-4 g-band nside=4096, DETMODEL
Gold 1.0.2-4 r-band nside=4096, DETMODEL
Gold 1.0.2-4 i-band nside=4096, DETMODEL
Gold 1.0.2-4 z-band nside=4096, DETMODEL
Gold 1.0.2-4 Y-band nside=4096, DETMODEL

MAG_MODEL

Gold 1.0.2-4 g-band nside=4096, MODEL
Gold 1.0.2-4 r-band nside=4096, MODEL
Gold 1.0.2-4 i-band nside=4096, MODEL
Gold 1.0.2-4 z-band nside=4096, MODEL
Gold 1.0.2-4 Y-band nside=4096, MODEL

MAG_APER_3

Gold 1.0.2-4 g-band nside=4096, APER_3
Gold 1.0.2-4 r-band nside=4096, APER_3
Gold 1.0.2-4 i-band nside=4096, APER_3
Gold 1.0.2-4 z-band nside=4096, APER_3
Gold 1.0.2-4 Y-band nside=4096, APER_3

MAG_APER_4

Gold 1.0.2-4 g-band nside=4096, APER_4
Gold 1.0.2-4 r-band nside=4096, APER_4
Gold 1.0.2-4 i-band nside=4096, APER_4
Gold 1.0.2-4 z-band nside=4096, APER_4
Gold 1.0.2-4 Y-band nside=4096, APER_4

Photo-z Training Files

DESDM photo-zs

ANNZ photo-zs

TPZ photo-z PDFs, probabilistic S/G and N_z

A complication for the calculation of photometric redshifts is the training fields. Most of the spectroscopic fields covered in SVA1 are SN fields and are thus more than full depth, which will bias any training and validation tests. Huan Lin has provided a separate reduction of the spectroscopic fields with coadditions of a subset of the finalcut data to approximate full survey depth. Keith is running SLR on these coadd catalogs to put them on the same footing as the rest of the SVA1 gold photometry, and then Huan (and others) will be able to do photo-z training. The training fields will be available here before New Year's, and the first of the photo-z catalogs should be appearing in early January.

Details

Initial Footprint Selection

The initial footprint was selected by way of Boris Leistedt's excellent SVA1 systematics maps. In particular, the "nccd" maps, which track the number of CCDs in each band (grizy) on an nside=2048 healpix grid. A minimum of 1 ccd was required in each of griz to go into the initial footprint. The total area covered in each band (before cutting bad regions, LMC, etc) is estimated to be:

band area (deg^2)
g 328.6
r 336.2
i 338.6
z 339.3
Y 303.9
griz 305.5

In addition to the basic footprint, a DEC>-61 cut was placed on the catalog to remove the LMC and R Doradus regions, which removes 44.5 deg^2. The removed region has proved very difficult to calibrate, and the high foregrounds limits their utility for extragalactic science.

Initial footprint, all with griz coverage of at least 1 ccd

In the end, we have a first estimate of the footprint with 0's and 1's on a healpix grid with nside=2048.

Star/Galaxy Separation

Star/Galaxy separation was performed via the modest star/galaxy classifier. This combines CLASS_STAR_I at the bright end, with SPREAD_MODEL_I and SPREADERR_MODEL_I to get a reasonably classification. The specific implementation is (with some gibberish pseudo-code):

(FLAGS_I <=3) AND NOT (((CLASS_STAR_I > 0.3) AND (MAG_AUTO_I < 18.0)) OR ((SPREAD_MODEL_I + 3*SPREADERR_MODEL_I) < 0.003) OR ((MAG_PSF_I > 30.0 AND MAG_AUTO_I < 21.0))))

And a crude star selection that avoids significantly negative spread_model values. (Take with a grain of salt: you'll want a magnitude cut as well for any calibration stars.):

(FLAGS_I <=3) AND (((CLASS_STAR_I > 0.3) AND (MAG_AUTO_I < 18.0) AND (MAG_PSF_I < 30.0) OR (((SPREAD_MODEL_I + 3*SPREADERR_MODEL_I) < 0.003) AND ((SPREAD_MODEL_I +3*SPREADERR_MODEL_I) > -0.003)))

The magnitude cut and class_star selections are there to ensure that the bright upturn in the non-homogenized coadds doesn't contaminate our galaxy selection.

Bad Region Selection

Following the original bad region selection , regions of bad photometry were found as follows. First, the catalog was pixelized with healpix, NSIDE=1024. The number of galaxies brighter than MAG_AUTO_I<22.5 in each pixel was counted, as well as the number of galaxies with "crazy colors" defined as DETMODEL_GMR < -1 OR DETMODEL_GMR > 4 OR DETMODEL_IMZ < -1 OR DETMODEL_IMZ > 4. These colors are wholly unphysical for relatively bright, well measured galaxies. High concentrations of galaxies with crazy colors implies that there is something wrong with the region. (Note that the "crazy color selection" was performed on DETMODEL magnitudes that had been corrected via SLR).

Number of galaxies in each pixel of nside=1024

The first cut made was on the galaxy density. This was especially important for the SN fields without nccd maps, to better define the edges. Looking at the above histogram, all pixels that had fewer than 18 galaxies brighter than 22.5 (3 sigma below the mean) were removed as "bad" pixels.

Histogram of fraction of galaxies with crazy colors in a pixel.

The next cut was made on the fraction of galaxies with crazy colors. A somewhat arbitrary cut of 10% was made. This removes only 1.5% of the area but grabs the bright airplanes, satellites, and regions of scattered light. However, it should be emphasized that depending on your desired science it is still recommended to further remove galaxies with "crazy colors". These are not highly clustered enough to make a material difference on the mask (well...this is the hope...) but they will still have odd photo-zs and other difficulties.

Each bad pixel with NSIDE=1024 is then split into 4 daughter pixels and these are set to 0 in the original nside=2048 footprint map. The final footprint with 254 deg^2 is:

Final footprint

Generation of Catalog

With the footprint in hand and bad regions masked out, the final value-added catalog files are now generated. All magnitudes are put on the same SLR footing using Keith's SLR v6. . The systematic error due to noise in the fits to the stellar locus is (2%?).

[need a plot here]

In addition, all fluxes are converted to picomaggies via a simple zeropoint shift:

fconv = 10**((30.0- (ZEROPOINT_X + ZP_SHIFT_X))/2.5)
flux_new = flux_old * fconv
fluxerr_new = fluxerr_old * fconv

Old Depth Maps

OLD

The masks have also been combined with Boris Leistadt's systematics maps (see Pixelizing And Modeling Mangle Masks) to estimate the depth appropriate for each magnitude, with SLR corrections, etc. These are good to ~3-4% (0.03-0.04 mag).

MAG_AUTO

Gold 1.0 g-band nside=4096, AUTO
Gold 1.0 r-band nside=4096, AUTO
Gold 1.0 i-band nside=4096, AUTO
Gold 1.0 z-band nside=4096, AUTO
Gold 1.0 Y-band nside=4096, AUTO

MAG_DETMODEL

Gold 1.0 g-band nside=4096, DETMODEL
Gold 1.0 r-band nside=4096, DETMODEL
Gold 1.0 i-band nside=4096, DETMODEL
Gold 1.0 z-band nside=4096, DETMODEL
Gold 1.0 Y-band nside=4096, DETMODEL

MAG_MODEL

Gold 1.0 g-band nside=4096, MODEL
Gold 1.0 r-band nside=4096, MODEL
Gold 1.0 i-band nside=4096, MODEL
Gold 1.0 z-band nside=4096, MODEL
Gold 1.0 Y-band nside=4096, MODEL

MAG_APER_3

Gold 1.0 g-band nside=4096, APER_3
Gold 1.0 r-band nside=4096, APER_3
Gold 1.0 i-band nside=4096, APER_3
Gold 1.0 z-band nside=4096, APER_3
Gold 1.0 Y-band nside=4096, APER_3

MAG_APER_4

Gold 1.0 g-band nside=4096, APER_4
Gold 1.0 r-band nside=4096, APER_4
Gold 1.0 i-band nside=4096, APER_4
Gold 1.0 z-band nside=4096, APER_4
Gold 1.0 Y-band nside=4096, APER_4