Project

General

Profile

SG separation challenge » History » Version 16

Ignacio Sevilla, 11/29/2013 09:27 AM

1 1 Ignacio Sevilla
h1. SG separation challenge
2 1 Ignacio Sevilla
3 1 Ignacio Sevilla
Now that several people are testing their own approaches:
4 1 Ignacio Sevilla
5 1 Ignacio Sevilla
* Cut-based with DESDM info (Eli, Diego, Nacho, Ryan...).
6 1 Ignacio Sevilla
* Multi-class (Maayane)
7 1 Ignacio Sevilla
* Random Forests (Ryan)
8 1 Ignacio Sevilla
* Boosted Decision Trees (Nacho)
9 1 Ignacio Sevilla
* Alternative Neural Network with probabilistic output (Chris Bonnett).
10 13 Basilio Santiago
* Probability based on spread model and photometry (DES-Brazil)
11 14 Basilio Santiago
* Others...
12 1 Ignacio Sevilla
13 1 Ignacio Sevilla
I think the time is right and the codes are mature to launch a specific SG separation challenge, mimicking the successful photo-z WG exercise.
14 1 Ignacio Sevilla
15 1 Ignacio Sevilla
We have to establish:
16 1 Ignacio Sevilla
17 1 Ignacio Sevilla
* The training/validation/testing sample (COSMOS, others).
18 6 Ignacio Sevilla
* Only stars and galaxies? What about QSOs, image artifacts?
19 3 Ignacio Sevilla
* The metrics (Fixed cut, Fixed purity, Fixed Efficiency, ROC -- see example below).
20 7 Eli Rykoff
* SVA1 systematics: correlations with depth, Galactic latitude, seeing, etc.
21 4 Ignacio Sevilla
* Who/how to run it.
22 8 Ignacio Sevilla
* Is there any gain combining them (a committee)?
23 1 Ignacio Sevilla
* The schedule.
24 9 Maayane Soumagnac
25 9 Maayane Soumagnac
h1. The metric
26 9 Maayane Soumagnac
27 9 Maayane Soumagnac
We suggest to use the same metric as in the DES star/galaxy separation (on simulation) paper (arXiv:1306.5236).
28 9 Maayane Soumagnac
29 10 Maayane Soumagnac
h2. Completeness and Purity provided by a given classifier
30 10 Maayane Soumagnac
31 9 Maayane Soumagnac
We define the parameters used to quantify the quality of a star/galaxy classifier. For a given class of objects, X (stars or galaxies), we distinguish the surface density of well classified ob jects, N_X , and the misclassified objects, M_X .
32 9 Maayane Soumagnac
33 9 Maayane Soumagnac
* The galaxy completeness c^g is defined as the ratio of the number of true galaxies classified as galaxies to the total number of true galaxies. 
34 9 Maayane Soumagnac
* The stellar contamination f_s is defined as the ratio of stars classified as galaxies to the total amount of ob jects classified as galaxies. 
35 9 Maayane Soumagnac
* The purity p^g is defined as 1-f_s
36 9 Maayane Soumagnac
37 11 Maayane Soumagnac
!{width:400px}metric.png! 
38 10 Maayane Soumagnac
39 10 Maayane Soumagnac
h2. Plots
40 10 Maayane Soumagnac
41 10 Maayane Soumagnac
Bellow are three different plots we suggest to use to assess the performances of each classifier.
42 10 Maayane Soumagnac
43 10 Maayane Soumagnac
h3. Histograms
44 10 Maayane Soumagnac
45 10 Maayane Soumagnac
Example, on simulations, from arXiv:1306.5236
46 11 Maayane Soumagnac
!{width:400px}histo.png!
47 10 Maayane Soumagnac
48 10 Maayane Soumagnac
h3. purity as a function of magnitude (for fixed completeness, the threshold/cut is let free)
49 1 Ignacio Sevilla
50 12 Eli Rykoff
!{width:800px}Emma.png!
51 10 Maayane Soumagnac
52 16 Ignacio Sevilla
!{width:400px}sg_separation_purity_vs_magauto_50.0_efficiency.png! !{width:400px}sg_separation_purity_vs_magauto_90.0_efficiency.png! 
53 16 Ignacio Sevilla
54 15 Ignacio Sevilla
h3. completeness as a function of magnitude (for fixed purity, the threshold/cut is let free )
55 15 Ignacio Sevilla
56 15 Ignacio Sevilla
!{width:400px}sg_separation_efficiency_vs_magauto_95.0_purity.png! !{width:400px}sg_separation_efficiency_vs_magauto_99.0_purity.png!