Project

General

Profile

SG separation challenge » History » Version 26

Alex Drlica-Wagner, 01/17/2014 06:05 PM

1 1 Ignacio Sevilla
h1. SG separation challenge
2 1 Ignacio Sevilla
3 1 Ignacio Sevilla
Now that several people are testing their own approaches:
4 1 Ignacio Sevilla
5 19 William Wester
* Cut-based with DESDM info (Eli, Diego, Nacho, Ryan, William...).
6 1 Ignacio Sevilla
* Multi-class (Maayane)
7 1 Ignacio Sevilla
* Random Forests (Ryan)
8 20 Alex Drlica-Wagner
* Boosted Decision Trees (Nacho, Alex)
9 1 Ignacio Sevilla
* Alternative Neural Network with probabilistic output (Chris Bonnett).
10 13 Basilio Santiago
* Probability based on spread model and photometry (DES-Brazil)
11 14 Basilio Santiago
* Others...
12 1 Ignacio Sevilla
13 1 Ignacio Sevilla
I think the time is right and the codes are mature to launch a specific SG separation challenge, mimicking the successful photo-z WG exercise.
14 1 Ignacio Sevilla
15 1 Ignacio Sevilla
We have to establish:
16 1 Ignacio Sevilla
17 1 Ignacio Sevilla
* The training/validation/testing sample (COSMOS, others).
18 17 Ignacio Sevilla
I have prepared a 70/30 training/testing with the deep COSMOS field matched to ACS imaging. About 280 parameters, up to each tester to choose which.
19 17 Ignacio Sevilla
Besides new datasets, also consider shallower COSMOS. Also consider fixed set of parameters as Eduardo suggests. Also need to add SLR corrections though I think not very important now.
20 6 Ignacio Sevilla
* Only stars and galaxies? What about QSOs, image artifacts?
21 17 Ignacio Sevilla
Star/galaxy for round 1.
22 3 Ignacio Sevilla
* The metrics (Fixed cut, Fixed purity, Fixed Efficiency, ROC -- see example below).
23 21 Ignacio Sevilla
I would prefer to use ROC, i.e., True Positive Rate vs False Positive Rate curve formed changing the threshold (thanks Alex for pointing out mistake in previous ROC!).
24 7 Eli Rykoff
* SVA1 systematics: correlations with depth, Galactic latitude, seeing, etc.
25 4 Ignacio Sevilla
* Who/how to run it.
26 17 Ignacio Sevilla
I suggest each group providing an output file with id (or ra,dec on first round) plus galaxy probability or binary value.
27 8 Ignacio Sevilla
* Is there any gain combining them (a committee)?
28 1 Ignacio Sevilla
* The schedule.
29 9 Maayane Soumagnac
30 18 Ignacio Sevilla
[[des-sci-verification:SG_separation_challenge_details|Test Details]]
31 18 Ignacio Sevilla
32 22 Alex Drlica-Wagner
h1. Comparison metrics
33 9 Maayane Soumagnac
34 25 Alex Drlica-Wagner
There are a number of metrics that can be used for comparing the performance of classifiers. Some especially useful metrics are those defined in the DES star/galaxy separation (on simulation) paper "arXiv:1306.5236":http://arxiv.org/abs/1306.5236 and the "receiver operating characteristic (ROC)":http://en.wikipedia.org/wiki/Receiver_operating_characteristic generally used for classifier comparison.
35 9 Maayane Soumagnac
36 10 Maayane Soumagnac
h2. Completeness and Purity provided by a given classifier
37 10 Maayane Soumagnac
38 22 Alex Drlica-Wagner
We define the parameters used to quantify the quality of a star/galaxy classifier. For a given class of objects, X (stars or galaxies), we distinguish the surface density of properly classified objects, N_X , and the misclassified objects, M_X .
39 9 Maayane Soumagnac
40 9 Maayane Soumagnac
* The galaxy completeness c^g is defined as the ratio of the number of true galaxies classified as galaxies to the total number of true galaxies. 
41 22 Alex Drlica-Wagner
* The stellar contamination f_s is defined as the ratio of stars classified as galaxies to the total amount of objects classified as galaxies. 
42 11 Maayane Soumagnac
* The purity p^g is defined as 1-f_s
43 10 Maayane Soumagnac
44 10 Maayane Soumagnac
!{width:400px}metric.png! 
45 10 Maayane Soumagnac
46 10 Maayane Soumagnac
Bellow are three different plots we suggest to use to assess the performances of each classifier.
47 10 Maayane Soumagnac
48 10 Maayane Soumagnac
h3. Histograms
49 10 Maayane Soumagnac
50 10 Maayane Soumagnac
Example, on simulations, from arXiv:1306.5236
51 11 Maayane Soumagnac
!{width:400px}histo.png!
52 10 Maayane Soumagnac
53 10 Maayane Soumagnac
h3. purity as a function of magnitude (for fixed completeness, the threshold/cut is let free)
54 1 Ignacio Sevilla
55 23 Alex Drlica-Wagner
!{width:600px}Emma.png!
56 1 Ignacio Sevilla
57 23 Alex Drlica-Wagner
!{width:300px}sg_separation_purity_vs_magauto_50.0_efficiency.png! !{width:300px}sg_separation_purity_vs_magauto_90.0_efficiency.png!
58 1 Ignacio Sevilla
59 1 Ignacio Sevilla
h3. completeness as a function of magnitude (for fixed purity, the threshold/cut is let free )
60 1 Ignacio Sevilla
61 24 Alex Drlica-Wagner
!{width:300px}sg_separation_efficiency_vs_magauto_95.0_purity.png! !{width:300px}sg_separation_efficiency_vs_magauto_99.0_purity.png!
62 22 Alex Drlica-Wagner
63 22 Alex Drlica-Wagner
h2. Receiver operating characteristics
64 22 Alex Drlica-Wagner
65 22 Alex Drlica-Wagner
The receiver operating characteristic (ROC) provides another tool for evaluating the performance of classifiers. The ROC provides some information orthogonal to that in the completeness vs purity plots:
66 22 Alex Drlica-Wagner
67 26 Alex Drlica-Wagner
* Because ROCs compare the true positive rate to the false positive rate, they do not depend on relative composition of the test sample. Thus, unlike the purity, they contain information only about the intrinsic performance of the classifier and not the test sample.
68 26 Alex Drlica-Wagner
* ROCs allow classifiers to be compared without requiring a threshold/cut to be placed on the output. This is useful because different projects possess different requirements on object sample, completeness, purity, etc. The area under the ROC can serve as a very high-level scalar metric for classifier performance.
69 26 Alex Drlica-Wagner
* Once a threshold/cut is placed, we can generate magnitude dependent true positive vs false positive rate plots which would be intrinsic to the classifiers.