Project

General

Profile

SG separation challenge details

Participants

Round 1

The goal for this round is to streamline this process. Results will not be so relevant.

Standardized inputs

Training and testing sets are based on matched COSMOS HST (ACS) imaging to DES and DECam community time images. They can be downloaded from this link:

http://wwwae.ciemat.es/~sevilla/catalogs/
(user: des, passwd: usual one)

The catalogs are round1_training_set.fits and round1_test_set.fits.

For details on how these catalogs were generated, check:

https://cdcvs.fnal.gov/redmine/projects/des-lss/wiki/Systematics_tests

specially the first section and the 'Star-galaxy separation' section.

The total area is around 1 sq.deg. There are a total of ~212000 objects, of which 70% (150000) can be used for training (train_sgchallenge_r1.fits file).

Known caveats:

  • this catalog is NOT SLR or galaxy locus corrected, however the effects on colors should be small from what I have seen in B.Armstrong's plots of the COSMOS field using synthetic star loci.
  • this catalog is deeper than standard DES depth, so we cannot extrapolate results to what we will really have in the main survey yet.
  • it is not known yet whether we have some QSOs in the point source classifications (see below).

Parameters

There are 224 DECam parameters and 60 ACS parameters (subscript _acs). ACS parameters cannot be used for training except the mu_class_acs parameter which provides the truth value we will use in this round. There are two additional parameters: cntr and Separation, coming from matching with TOPCAT, that should be disregarded too.

mu_class_acs = 1 GALAXIES
mu_class_acs = 2 POINT SOURCES
mu_class_acs = 3 Fake ACS detections (IGNORE)
mu_class_acs = -999 Candidate Fake DECam detections (IGNORE)

Note that in the mu_class_acs=2 case, I have checked against zCOSMOS that these objects have redshift = 0 but only those that have a zCOSMOS redshift. That means that some QSOs might be creeping in.

Any of the 226 DECam parameters or combination of them can be used. This includes colors, mag_psf-mag_model, spread_model and class_star too! We might want to homogeneize or streamline a little bit for future rounds.

Round 1 parameter list

It is not mandatory to use the whole training set or all of the parameters. We can tune that later on.

Standardized outputs.

After training your classifier (if it is learning based), please run your code on the test_sgchallenge_r1.fits file.

Provide compressed ASCII file with RA, DEC, Separator value

In future rounds we will use COADD_OBJECTS_ID

Separator value can be:

  • Binary: 0 for galaxies and 1 for stars (e.g. choice after a threshold is set)
  • Probabilistic or pseudo-probabilistic: Ranging from 0 (galaxies) to 1 (stars) (e.g. CLASS_STAR).
  • Direct classifier output: Any real number. In that case, please provide a typical range for possible thresholds to make things quicker. (e.g. SPREAD_MODEL, range from -0.003 to +0.01).

The files can be provided to to generate a few plots for this testing round. Alternatively, you can produce your own plots, but we would prefer to have a more homogeneous report.

Results (Jan 28 2014)

The results for the range 16<MAG_AUTO_I<26; 19.5<MAG_AUTO_I<20.5; 23<MAG_AUTO_I<24 are shown below.

16<MAG_AUTO_I<26 19.5<MAG_AUTO_I<20.5 23<MAG_AUTO_I<24

Here "modest" is simply spread_model+3*spreaderr_model.
Here "bayesian" uses the product pg_col*pg_i*pg_sm (see documentation).

The area under the curve for the first plot are:

Classifier ROC area Completeness at 95% purity Completeness at 99% purity Purity at 95% completeness
SPREAD_MODEL 0.905 94.7% 73.4% 95%
CLASS_STAR 0.886 99.6% - 96.7%
MODEST 0.935 99.5% 71.8% 98%
TPZ 0.937 99.8% 71.7% 97.7%
BAYESIAN 0.923 >99.2% 45.3% 96.8%
BDT 0.967 >99.3% 93.7% 98.2%
SKYNET 0.923 99.9% 53.3% 98.0%
What can we measure beyond this?
  • More binning in magnitude.
  • As a function of photo-z.
  • As a function of seeing, airmass, extinction... can be treated with cross-correlations of maps for LSS. But maybe needs specific study for other science.
  • Spatial correlations (statistical performance metrics, truth tables on the scale of degrees?).
  • Are failures and outliers the same in all codes? Can a 'voting' scheme improve the results?
  • The impact of field density on classifier performance (de-blending issues)

Round 2

The goal for this round is to start getting a fairer comparison of the different classifiers:

  • We have expanded to the imaging and spectroscopic fields detailed below.
  • We have decided not to make any common 'pre-treatment' of data and leave that to each code owner.
  • No QSO selection yet (Richard McMahon and the QSO WG will have some catalogs ready in a few weeks).
  • 60/20/20 training/validation/testing samples.
  • Star/galaxy ratios will vary across the samples as per their magnitude.

Essentially it is round 1, with as many additional spectroscopic info as possible, some modifications in the star/galaxy ratio of the training set and making the testing set blind.

Results I will compile:
  1. The ROC curves for those classifiers that provide a real-numbered output, for the whole set and the three different magnitude bins, which have a variable star/galaxy ratio from 30% to <5%.
  2. Galaxy Completeness and Purity values as the table for round 1
  3. Star Completeness and Purity for magnitude < 21. This is a special case to look for good classifiers for bright stars.

Standardized inputs

The following table includes the fields matched to the SVA1-Gold catalog and included in the provided files.

Catalog Field Area Type Nb. of good quality stars Magnitude range Associated paper Comments
ACS COSMOS COSMOS ~1 deg^2 Space imaging ~15000 mag_auto_i<25 Leauthaud07
ACS GOODS-S SN-C ~160 arcmin^2 Space imaging ~300 mag_auto_z < 27 Giavalisco04 Conservative cut in mag_auto<24
VVDS-DEEP-02 SN-X 0.6 deg^2 Spectroscopy ~600 17.5<mag_auto_i<24.75 LeFèvre13 Includes ultra-deep
VVDS-CDFS SN-C Spectroscopy ~100 17.5<mag_auto_i<24
ACES SN-C Spectroscopy ~300 18<mag_auto_i<23 Cooper12
VIPERS-W1 SN-X Spectroscopy ~900 mag_auto_i<22.5 Guzzo13

Left aside for future rounds (TBC):

Catalog Field Area Type Nb. of good quality stars Magnitude range Associated paper Comments
SDSS Stripe 82 SN-S Spectroscopy TBC TBC
ACS CANDELS UDS SN-X Space imaging ~800? mag_auto_i < 29.5 Galametz13 NOT USED IN ROUND 2
zCOSMOS COSMOS Spectroscopy ~300 13.5<I_AB<22.5 Included in ACS COSMOS already
SuperCOSMOS All-sky? Proper Motion ~10000 < 19.5 Rowell & Hambly, 2011 Needs a closer look
COMBO-17 CDFS SN-C 30'x30' 17-Band Photometry ~1000 mag_r < 26 Wolf et al., 2004 SDSS-II used different COMBO-17 field
CFHTLenS S82 ~3 deg^2 Deep Ground-Based r < 26 Erben et al., 2012

Training and testing sets can be downloaded from this link:

http://wwwae.ciemat.es/~sevilla/catalogs/
(user: des, passwd: usual one)

The file is round2.tgz. In this file, you will find a training set (60%), a 'visible' testing set (aka validation set, 20%), and a 'blind' testing set (20%). There are four versions of each of these sets, corresponding to different magnitude ranges, and therefore twelve files in total.

Integral sets (all matched objects to Gold, 297k objects, around 93% from ACS-COSMOS)
round2_test_blind_set.fits
round2_test_visible_set.fits
round2_training_set.fits
Sets in MAG_AUTO_I bins (up to MAG_AUTO_I<24, 154k objects, around 87% from ACS-COSMOS)
round2_training_set_auto_i_lt_21.fits
round2_training_set_auto_i_21_23.fits
round2_training_set_auto_i_23_24.fits
round2_test_visible_set_auto_i_lt_21.fits
round2_test_visible_set_auto_i_21_23.fits
round2_test_visible_set_auto_i_23_24.fits
round2_test_blind_set_auto_i_lt_21.fits
round2_test_blind_set_auto_i_21_23.fits
round2_test_blind_set_auto_i_23_24.fits

Caveats of the ACS-COSMOS data:
  • ACS-COSMOS field is deeeper than standard DES depth, so we cannot extrapolate results to what we will really have in the main survey yet. I cut in mag_auto_i<24 but that is still not exactly like what we will have in DES. We need to reduce the COSMOS data to DES-like depths for future rounds.
  • There are some QSOs in the point source classifications in ACS-COSMOS.
Caveats of the spectroscopic matches:
  • Possible selection bias except for VVDS-02 and VVDS-CDFS, which were sampled without galaxy target selection.

Parameters

There are 261 columns in the files. All of them, except true_class and catalog_id, can be used by the algorithms. Combinations can be used to, e.g., colors, mag_psf-mag_model, spread_model and class_star too!

true_class is:
0 -- for galaxies
1 -- for stars

catalog_id is:
10 -- ACS-COSMOS
11 -- VVDS-02
12 -- VVDS-CDFS
13 -- VIPERS-W1
14 -- ACES
15 -- H-GOODS-S

The rest of the catalog parameters will be detailed ASAP at Round 2 parameter list.

It is not mandatory to use the whole training set or all of the parameters.

Standardized outputs.

After training and tuning your classifier using the training and/or visible test sets, please run your code on the blind test files. Provide ASCII files with the results of the latter with two columns: COADD_OBJECT_ID, Separator value

Separator value can be:

  • Binary: 0 for galaxies and 1 for stars (e.g. choice after a threshold is set)
  • Probabilistic or pseudo-probabilistic: Ranging from 0 (galaxies) to 1 (stars) (e.g. CLASS_STAR).
  • Direct classifier output: Any real number. In that case, please provide a typical range for possible thresholds to make things quicker. (e.g. SPREAD_MODEL, range from -0.003 to +0.01).

NOTE that for ROC curves we need a real valued output to be able to vary the threshold and draw the curve.

The files can be provided to .

HARD DEADLINE: March 7th 2014

Results (March 13 2014)

MAG_AUTO_I<21 21<MAG_AUTO_I<23 23<MAG_AUTO_I<24

Results are given below for the three bins:

Classifier ROC area Completeness at 95% purity Completeness at 99% purity Purity at 95% completeness
SPREAD_MODEL 0.9891/0.9441/0.8912 98.97/99.58/>99.38 97.69/<49/72.61 99.27/97.37/96.53
CLASS_STAR 0.9771/0.8861/0.7859 95.89/97.12/99.80 81.78/-/- 96.14/96.01/96.61
SPREAD_MODEL+3*error 0.9897/0.9457/0.8841 99.04/99.77/99.82 97.91/<60/68.72 99.27/98.36/96.96
TPZ 0.9954/0.9812/0.9460 99.52/99.96/99.99 98.64/93.35/91.84 99.57/99.84/98.78
RANDOM FORESTS 0.9960/0.9796/0.9368 99.56/99.96/99.99 99.08/94.20/91.29 99.05/98.95/98.76
BDT 0.9953/0.9726/0.9449 99.48/99.93/99.99 98.83/93.2/94 99.46/98.90/98.93
SVM 0.9904/0.9494/0.9002 98.97/99.63/99.94 93.98/75/71.67 98.78/97.69/98.05
SKYNET 0.9917/0.9739/0.9328 99.34/99.92/99.99 98.79/95.04/92.52 99.46/99/98.90

Modest class performance as a function of photo-z

Choosing a 98% purity cut, for the other classifiers we have:

Validation Ideas for SVA1

  • Compare the number of stars (galaxies) as a function of magnitudes (colors)
    • This would be a single plot with O(3) histograms comparing the number of objects output from each classifier
    • We can use various statistical methods (e.g. KS test) to test the statistical consistency of the histograms
  • Compare the spatial number density of stars (galaxies) as produced from each classifier.
    • The idea here would be to create spatial residual maps to find regions where the performance of the classifiers differs
      • This could be done simply by creating healpix maps reflecting the number of objects in each bin
      • We would expect regions of large residuals to be correlated with regions of bad seeing, PSF estimation, etc.
    • When comparing various classifiers, variations in the magnitude limit should cancel out
    • We may consider creating power spectra from two-point cross correlations between classifier catalog maps
  • For each individual classifier correlate the star and galaxy distributions.
    • The idea here would be to create a two-point cross-correlated power spectrum from star and galaxy healpix maps
    • Here it would be necessary to correct for depth.
    • Can use the SVA1 magnitude limit healpix map (from mangle maps)

Various round 2 classifier have been run on the full SVA1 Gold v1.0 catalog and collected here:
SVA1 Gold v1.0 SG Validation

BALROG Performance

Some preliminary single epoch tests have been run by Jelena Aleksic using BALROG (real images with embedded simulated objects). Her results can be found here. In the future, it would be interesting to run this on the coadds for greater similarity to SVA1.

Variable Combinations

In the previous rounds, classifiers were fed raw input variables without much pre-processing. Ideally, the classifiers should be robust enough to create powerful variable combinations from the raw input. However, it may be more robust to create a subset of physically motivated variable combinations so that the amount of free reign allowed to the classifier is decreased. Some early ideas have been put together on a page here here.

Shallow Coadds for SG Separation

Investigating the use of Eli's SVA1_SN_Field_Shallow_Coadds for SG separation.

Classifiers comparison

Purity (galaxies) at 88.9% completeness

Purity (stars) at the fixed completenesse, for all classifiers.

Round 3

The goal for this round is to arrive to a set of well-studied classifiers which can be incorporated to the SVA1 catalog (and eventually survey catalogs).

  • We have expanded the spectroscopic data as detailed below.
  • We have substituted COSMOS by Eli's T1000 reduction of the same field (same depth, different number of exposures though).
  • A training set and un-blinded test set is provided. Each code-owner will then run the code on the blind set. Training can be done per magnitude bin or not. Additionally, we request that a training/testing run is done ONLY on COSMOS field, by selecting FIELD==10, and providing those results on the blind set with FIELD==10.

Parameters

RA, DEC
MAG_MODEL_GRIZ and MAGERR errors
MAG_DETMODEL_GRIZ and MAGERR errors
MAG_AUTO_GRIZ and MAGERR errors
MAG_PSF_GRIZ and MAGERR errors
SPREAD_MODEL_GRIZ and SPREADERR errors
CLASS_STAR_GRIZ (not in Gold, taken from SVA1_COADD_OBJECTS)
FWHM_WORLD_GRIZ (not in Gold, taken from SVA1_COADD_OBJECTS)
MODEST_CLASS
To be added (not in DB, need to extract them from flat files):
A_IMAGE_GRIZ
B_IMAGE_GRIZ
KRON_RADIUS_GRIZ

TRUE_CLASS is:
0 -- for galaxies
1 -- for stars
FIELD is:
10 -- ACS-COSMOS
11 -- VVDS-02
12 -- VVDS-CDFS
13 -- VIPERS-W1
14 -- ACES
15 -- H-GOODS-S
16 -- SDSS-DR10
17 -- GAMA
18 -- DES-AAOmega
19 -- UDS

Standardized inputs

The following table includes the fields matched to the SVA1-Gold catalog and included in the provided files.

Catalog Field Area Type Nb. of good quality stars Magnitude range Associated paper Comments
ACS COSMOS COSMOS ~1 deg^2 Space imaging ~15000 mag_auto_i<25 Leauthaud07
VVDS-DEEP-02 SN-X 0.6 deg^2 Spectroscopy ~600 17.5<mag_auto_i<24.75 LeFèvre13 Includes ultra-deep
VVDS-CDFS SN-C Spectroscopy ~100 17.5<mag_auto_i<24
ACES SN-C Spectroscopy ~300 18<mag_auto_i<23 Cooper12
ACS GOODS-S SN-C ~160 arcmin^2 Space imaging ~300 mag_auto_z < 27 Giavalisco04 Conservative cut in mag_auto<24

The following table includes the fields that have been added from the OzDES project (DES-doc-7548, as of Sep.15th 2014), matched to the SVA1-Gold catalog and included in the provided files.

Catalog Field Area Type Nb. of good quality stars Magnitude range Associated paper Comments
SDSS-DR10  SN-X  30 deg^2  Spectroscopy  ~1000 14-22.5   Ahn13
VIPERS SN-X TBD Spectroscopy ~250 17-23 Guzzo13
GAMA SN-X 10 deg^2 Spectroscopy ~140 14-20 Liske14
DES-AAOmega SN-X, SN-C, SN-E TBD Spectroscopy ~620 15-22 TBD
UDS SN-X TBD Spectroscopy ~20 18-26 TBD

Note that some regions of the OzDES compilation were excluded, either due to not being reliable, according to the documentation (NED), or not containing stars (2dF, 6dF, SNLS) to avoid overpopulating with even more galaxies the training sample. In total the spectroscopic catalogs include a little more than 3000 stars.

The spectroscopic catalogs have been matched positionally (1") to SVA1_GOLD objects (and SVA1_COADD_OBJECTS to retrieve additional columns missing in Gold). Cuts have been made in -1<(mag_auto_g-mag_auto_r)<4, -1<(mag_auto_i-mag_auto_z)<4, niter_model_i>0, mag_psf_i<30, flags_i<4. The resulting catalog contains ca. 41000 objects, from which some duplicate objects have been removed (two spectra from different catalogs for the same COADD_OBJECT_ID object).

In addition to this, the T1000 shallow coadds (matched to ACS COSMOS catalog) is used.

Training and testing sets can be downloaded from this link:

http://wwwae.ciemat.es/~sevilla/catalogs/
(user: des, passwd: usual one)

The file is round3.tgz. In this file, you will find a training set (60%), a 'visible' testing set (aka validation set, 20%), and a 'blind' testing set (20%). There are four versions of each of these sets, corresponding to different magnitude ranges, and therefore twelve files in total (112k objects, around 63% from ACS-COSMOS).

*Integral sets
round3_test_blind_set.fits --> provide your results on this one
round3_test_visible_set.fits
round3_training_set.fits
*Sets in MAG_AUTO_I bins (optional to use, separated here for convenience)
round3_training_set_auto_i_lt_21.fits
round3_training_set_auto_i_21_23.fits
round3_training_set_auto_i_23_24.fits
round3_test_visible_set_auto_i_lt_21.fits
round3_test_visible_set_auto_i_21_23.fits
round3_test_visible_set_auto_i_23_24.fits
round3_test_blind_set_auto_i_lt_21.fits
round3_test_blind_set_auto_i_21_23.fits
round3_test_blind_set_auto_i_23_24.fits

Remember to train on both the whole training set and again using only COSMOS (FIELD 10).

Standardized outputs.

Submission Guidelines for performance plots on blind sample:

After training and tuning your classifier using the training and/or visible test sets (both on full sets AND selecting FIELD10 for COSMOS only tests), please run your code on the blind test files (whole set AND FIELD == 10 only). Provide 2 ASCII files with the results of the latter with two columns: BLIND_ID, Separator value

Separator value can be:

  • Probabilistic or pseudo-probabilistic: Ranging from 0 (galaxies) to 1 (stars) (e.g. CLASS_STAR).
  • Direct classifier output: Any real number. In that case, please provide a typical range for possible thresholds to make things quicker. (e.g. SPREAD_MODEL, range from -0.003 to +0.01).

The files can be provided to .

For the SVA1 validation tests, please apply your training (with and without COSMOS) on the SVA1 CATALOG
For this part, make the results available here in the table (as with round 2) The submission guidelines for Alex's scripts are as follows:

Submission Guidelines for SVA1 validation:
  • Classifier output should be submitted in FITS file format
  • Each FITS file should contain two columns: (1) COADD_OBJECTS_ID (64-bit int 'K') (2) CLASS_OUTPUT (4-byte float 'E')
  • The COADD_OBJECTS_ID should match exactly (content and order) with that in sva1_gold_1.0.2_catalog_basic.fits
  • CLASS_OUTPUT should be a real-valued number in the range [0,1]
  • Objects with a more galaxy-like classification should have output values closer to 0, while star-like objects should be closer to 1. (This is an arbitrary choice be follows CLASS_STAR and seems to be the convention for most submissions.)

Note that if you find out in the training and test sets, that your classifier works better with CLASS_STAR, FWHM_WORLD, A_IMAGE, B_IMAGE or KRON_RADIUS, we have to wait to include those in SVA1_GOLD.

HARD DEADLINE: October 17th 2014

Results (October 23 2014)

After processing the submissions from Christopher Bonnett (random forests, boosted decision trees, Skynet), Alex Drlica-Wagner (boosted decision trees), Edward Kim (TPZ-like code) and Maayane Soumagnac (ANNz-like code) and comparing against DESDM classifier methods, results indicate that:

  • Machine Learning methods are better perfomant than DESDM classifiers also with this new training set on a test set with the same characteristics. In particular, random forest methods seem to be doing a little bit better.
  • The particular choice that Chris has included using mag_psf-mag_model (aka SDSS concentration parameter) gives somewhat better results.
  • Using colors instead of just magnitudes does not change the results.
  • 99% overall galaxy purities are achievable with minor hit on completeness, as compared to modest.
  • Work is needed to improve star selection at the faintest end (i~23.5), where the overwhelming number of galaxies tend to classify everything as such (maybe use photo-z info here? or exclusively color and NOT magnitude? different training parameters as a function of magnitude...)
  • About a 30% net gain in correctly identifying stars out of the galaxy sample is achieved going from MODEST to TPZSG (according to the calibration sample).
  • Validation on SVA1 area
Pending (this week):
  • Completeness and purity plots (have to send Maayane the files, Nacho's fault).
  • Star purity and completeness.
  • Create catalog based on this round for SVA1 science use (Matías, Edward). DONE
  • Update results with new TPZ runs, and possibly other submissions in SVA1 area. DONE for TPZ
Longer term:
  • Combination of methods.
  • Incorporate a Bayesian method.
  • Morphology only classifiers
  • Color only classifiers

Key to different classifier runs

Classifier name Input parameters
TPZ_MAG mag_psf and mag_model all bands
TPZ_CLR mag_psf , mag_model , det_model colors all bands
TPZ_PSF mag_psf , mag_model , det_model colors, mag_psf-mag_model all bands
BDT_MORPH spread_model, spreaderr_model all bands
BDT_AUTO mag_auto, spread_model, spreaderr_model all bands
MULTICLASS_1
MULTICLASS_2
MULTICLASS_3
{RF,BDTM,SKYNET}_COLOR colors for auto and det_model
{RF,BDTM,SKYNET}_COLOR_1MAG colors for auto and det_model and mag_auto_r
{RF,BDTM,SKYNET}_COLOR_MAG colors and mags for auto and det_model
{RF,BDTM,SKYNET}_COLOR_PSF psf,auto,det model color and mags and psf_mag - model mag

ROC curves

Completeness and purity tables

Classifier ROC area Completeness at 95% purity Completeness at 99% purity Purity at 95% completeness
SPREAD_MODEL 0.926 95.19 78.23 95.29
CLASS_STAR 0.907 94.96 67.00 94.95
SPREAD_MODEL+3*error 0.926 97.49 <82 97.54
TPZ_MAG_COSMOS 0.969 98.42 94.74 98.94
TPZ_CLR_COSMOS 0.968 98.42 94.24 98.88
TPZ_MAG_ALLFIELDS 0.979 99.91 95.41 99.04
TPZ_CLR_ALLFIELDS 0.976 99.79 94.67 98.92
BDT_SHAPE_COSMOS 0.970 99.79 93.14 98.70
BDT_AUTO_COSMOS 0.970 98.63 93.66 98.79
BDT_SHAPE_ALLFIELDS 0.961 99.29 87.28 97.41
BDT_AUTO_ALLFIELDS 0.973 99.67 90.28 98.40
MULTICLASS_3 TBD 99.74 92.22 98.80
RF_ALL_COLOR 0.976 99.74 93.11 98.66
RF_ALL_COLOR_1MAG 0.977 99.03 94.73 98.96
RF_ALL_COLOR_MAG 0.975 98.65 95.00 99.00
RF_ALL_PSF 0.988 99.92 97.00 99.34
RF_COSMOS_COLOR 0.976 99.68 93.71 98.80
RF_COSMOS_COLOR_1MAG 0.50?
RF_COSMOS_COLOR_MAG 0.976 98.65 94.86 99.01
RF_COSMOS_PSF 0.975 98.66 94.55 98.94
BDTM_ALL_COLOR 0.976 99.78 92.71 98.71
BDTM_ALL_COLOR_1MAG 0.974 98.87 94.21 98.81
BDTM_ALL_COLOR_MAG 0.972 98.51 94.49 98.89
BDTM_ALL_PSF 0.984 99.96 96.35 99.17
BDTM_COSMOS_COLOR 0.976 99.75 92.76 98.71
BDTM_COSMOS_COLOR_1MAG 0.975 99.15 93.65 98.78
BDTM_COSMOS_COLOR_MAG 0.971 98.54 94.24 98.88
BDTM_COSMOS_PSF 0.971 98.37 94.47 98.86
SKYNET_ALL_COLOR 0.975 99.74 93.28 98.84
SKYNET_ALL_COLOR_1MAG 0.953 97.43 93.43 98.84
SKYNET_ALL_COLOR_MAG 0.968 98.50 94.32 98.85
SKYNET_ALL_PSF 0.973 99.74 94.66 98.99
SKYNET_COSMOS_COLOR 0.975 99.64 92.05 98.72
SKYNET_COSMOS_COLOR_1MAG 0.953 98.20 90.51 98.69
SKYNET_COSMOS_COLOR_MAG 0.952 98.01 90.93 98.66
SKYNET_COSMOS_PSF 0.961

These ML methods tend to classify all true stars near the magnitude limit as galaxies. For round 2, the effect was not so important, but happened somewhat beyond the N(m) turnover. With a very harsh cut, you can get rid of them, but this means that the outputs may not be trulyreflecting a 'probability', as there are few objects in 'uncertain' regions (as happened with CLASS_STAR, where these objects were dumped in the ~0.5 region).

The TPZ output for true stars in the blind sample:

The true distributions of objects in the blind sample:

Magnitude distribution of missclassified objects:

Output for true stars and galaxies, with the N(m) distribution overplotted, for different mixtures of COSMOS data in the training (thanks Edward!):

Classifiers comparison

Purity (galaxies) at 88.9% and 96.0% completeness

Purity (stars) at the fixed completenesse, for all classifiers.

Other ideas for future experimentation

  • Add photo-zs as additional column (for training and/or reporting results as a function of it).
  • Adding multi-epoch based information (spread_model median or at best fwhm (Anne B); size parameter (Erin S))
  • Use common 'super-parameters' (after PCA done in Multiclass for instance) as inputs for ML methods.* Vista or other near-IR catalogs
  • Add QSO catalog by QSO WG.
  • Add artifacts in truth table.
  • Stripe 82 matches.
  • Old reductions by Fermilab (Huan) in VVDS-F10 and VVDS-F14.
  • Should we consider using CFHT imaging?
  • Proper motion catalogs of bright stars over the full DES footprint
  • Rich stellar regions surrounding the LMC
  • Physically motivated variable combinations
  • Re-weighting training variable distributions to match SVA1 distributions
  • Validations on UFIG and BALROG simulations