SG separation challenge » History » Version 16

« Previous - Version 16/37 (diff) - Next » - Current version
Ignacio Sevilla, 11/29/2013 09:27 AM

SG separation challenge

Now that several people are testing their own approaches:

  • Cut-based with DESDM info (Eli, Diego, Nacho, Ryan...).
  • Multi-class (Maayane)
  • Random Forests (Ryan)
  • Boosted Decision Trees (Nacho)
  • Alternative Neural Network with probabilistic output (Chris Bonnett).
  • Probability based on spread model and photometry (DES-Brazil)
  • Others...

I think the time is right and the codes are mature to launch a specific SG separation challenge, mimicking the successful photo-z WG exercise.

We have to establish:

  • The training/validation/testing sample (COSMOS, others).
  • Only stars and galaxies? What about QSOs, image artifacts?
  • The metrics (Fixed cut, Fixed purity, Fixed Efficiency, ROC -- see example below).
  • SVA1 systematics: correlations with depth, Galactic latitude, seeing, etc.
  • Who/how to run it.
  • Is there any gain combining them (a committee)?
  • The schedule.

The metric

We suggest to use the same metric as in the DES star/galaxy separation (on simulation) paper (arXiv:1306.5236).

Completeness and Purity provided by a given classifier

We define the parameters used to quantify the quality of a star/galaxy classifier. For a given class of objects, X (stars or galaxies), we distinguish the surface density of well classified ob jects, N_X , and the misclassified objects, M_X .

  • The galaxy completeness c^g is defined as the ratio of the number of true galaxies classified as galaxies to the total number of true galaxies.
  • The stellar contamination f_s is defined as the ratio of stars classified as galaxies to the total amount of ob jects classified as galaxies.
  • The purity p^g is defined as 1-f_s


Bellow are three different plots we suggest to use to assess the performances of each classifier.


Example, on simulations, from arXiv:1306.5236

purity as a function of magnitude (for fixed completeness, the threshold/cut is let free)

completeness as a function of magnitude (for fixed purity, the threshold/cut is let free )