Sub-Pixelizing Mangle Masks » History » Version 12

Eli Rykoff, 02/04/2014 11:02 AM

1 1 Eli Rykoff
h1. Sub-Pixelizing Mangle Masks
2 1 Eli Rykoff
3 1 Eli Rykoff
4 1 Eli Rykoff
5 1 Eli Rykoff
Currently, we have the full mangle mask which gives the 2" aperture depth at every point in the survey, and the healpix nside=4096 approximation of the mangle masks.  The full mangle mask is ... unwieldy, which it would be nice to use a pixelized version.  Note that this is close to the finest practical pixelization before desktop workstations will struggle with the memory requirements.
6 1 Eli Rykoff
7 1 Eli Rykoff
However, as currently implemented, the pixelization simply takes the depth at the central point of that pixel.  As each pixel at nside=4096 is 0.75 arcmin^2, this has trouble around stars, boundaries, etc.
8 1 Eli Rykoff
9 1 Eli Rykoff
h2. The Subpixel Method
10 1 Eli Rykoff
11 1 Eli Rykoff
A few of us (Alex D-W in particular) have been working on seeing what can be done to calculate the depth on a finer scale and then average it.  Note that in order for this to work we need 2 output maps: the first has the average depth in the pixel *where the image was not completely masked*, and the second is the fraction of the pixel area that is not masked.  In this way we can create an "anti-aliased": map that contains most of the information from the fine scales but is more practical to use.  (In particular, this format is useful for cluster masked-region corrections.)
12 1 Eli Rykoff
13 11 Eli Rykoff
The reason that Aurelien has been doing a single-pixel value is that the regular mangle code is too slow for this procedure (which is optimized for other things.)  Luckily, Erin Sheldon has adapted some super-fast code from Martin White in the "pymangle": python package that is over 50x faster than the default mangle code for looking up weights ... this is very useful!
14 1 Eli Rykoff
15 3 Eli Rykoff
h2. The Reference
16 3 Eli Rykoff
17 3 Eli Rykoff
Although doing the full survey at very large nside is unwieldy, we can do a small region at larger nside without too much trouble (though we have to avoid trying to use the regular healpix map functions or you'll blow away your RAM.)  As a reference, I have started with the i-band mask in the field of RXJ2248, and have run healpix with nside = 65536 (this is 256x more pixels than nside=4096).
18 3 Eli Rykoff
19 4 Eli Rykoff
The map looks like this at very high resolution, along with a zoom in to a section in the center:
20 1 Eli Rykoff
21 1 Eli Rykoff
22 6 Eli Rykoff
23 6 Eli Rykoff
It's also worth looking at how the map changes as we go from high to low resolution:
24 6 Eli Rykoff
25 7 Eli Rykoff
26 1 Eli Rykoff
27 4 Eli Rykoff
h2. The Subpixel Tests
28 1 Eli Rykoff
29 4 Eli Rykoff
For a first test, I have taken the i-band mask in the field of RXJ2248, and run nside=4096 with 1, 4, 16, 64, and 256 sub-pixels.  In practice, to do all of SVA1 (or the DES footprint), 256 subpixels is "doable" but significantly slower.  If we can get away with 16 or 64 that would make things run a lot faster.  (The run time for the whole lot on this field was 20 minutes, about half of that for the 256 subpixel run).  
30 4 Eli Rykoff
31 8 Eli Rykoff
These runs are done at fine scale and then averaged down to the coarser nside=4096 scale.  I look at the output weight and fraction-observed maps.  The delta-mag histogram is shown for the nside=4096 subpixelized version relative to the high res (nside=65536) version.  I calculate the fraction of "bad" pixels where the weight is misestimated by 0.1 mag or greater.  There is also a histogram of the bad fraction relative to the high res run, and I note when the bad fraction is off by >5%.  These are somewhat arbitrary cuts, but they get to the rate of bad outliers.  
32 8 Eli Rykoff
33 4 Eli Rykoff
34 1 Eli Rykoff
h3. 256 Subpixels
35 1 Eli Rykoff
36 2 Eli Rykoff
37 2 Eli Rykoff
38 4 Eli Rykoff
39 2 Eli Rykoff
40 1 Eli Rykoff
h3. 64 Subpixels
41 2 Eli Rykoff
42 2 Eli Rykoff
43 2 Eli Rykoff
44 4 Eli Rykoff
45 1 Eli Rykoff
46 2 Eli Rykoff
h3. 16 Subpixels
47 2 Eli Rykoff
48 2 Eli Rykoff
49 2 Eli Rykoff
50 4 Eli Rykoff
51 2 Eli Rykoff
52 2 Eli Rykoff
h3. 4 Subpixels
53 2 Eli Rykoff
54 2 Eli Rykoff
55 1 Eli Rykoff
56 4 Eli Rykoff
57 2 Eli Rykoff
58 2 Eli Rykoff
h3. 1 (sub)pixel
59 2 Eli Rykoff
60 2 Eli Rykoff
61 2 Eli Rykoff
62 4 Eli Rykoff
63 2 Eli Rykoff
64 10 Eli Rykoff
h2. Higher Resolutions?
65 10 Eli Rykoff
66 10 Eli Rykoff
NSIDE=4096 is close to the highest resolution that is practical to store a full healpix map of the sky such that many of the healpix tools (mollview, for example) will work.  However, significant data compression can be achieved by storing a list of pixel numbers and weights.  What are the practical limitations for a 5000 deg^2 survey?
67 10 Eli Rykoff
68 10 Eli Rykoff
Well, storing pixel numbers (64 bit int) and weights (32 bit float) for 5000 deg^2 at nside=65536 would take 200 Gb per band.  This is non-trivial (though in 5 years...?)  Right now it would take 12 Gb/band for all of SVA1.  Doing nside=32768 gets us down by a factor of 4 (50Gb/band).  This actually may be practical, and as seen below only fails for about 1.5% of the area.  
69 10 Eli Rykoff
70 10 Eli Rykoff
Meanwhile, just to be clear, the polygon files currently take ~5Gb/band for SPT-E (150 deg^2), extrapolating to 167 Gb/band for the full survey.  So nside=32768 would be a significant savings over the full mangle mask, and is also more efficient to use (for some purposes at least).
71 10 Eli Rykoff
72 10 Eli Rykoff
73 10 Eli Rykoff
74 10 Eli Rykoff
75 12 Eli Rykoff
h2. Higher Resolution residual maps
76 12 Eli Rykoff
77 12 Eli Rykoff
These are what the residuals look like compared to hi-res if you take the center value of each pixel:
78 12 Eli Rykoff
79 12 Eli Rykoff
80 12 Eli Rykoff
81 12 Eli Rykoff
82 12 Eli Rykoff
83 12 Eli Rykoff
84 2 Eli Rykoff
h2. Summary
85 9 Eli Rykoff
86 1 Eli Rykoff
In terms of getting things roughly right, things actually converge quickly.  As soon as we average over 16 subpixels we're at the limit of how well we can do with an nside=4096 map.  However, it is apparent that nside=4096, even with subpixel averaging, is insufficient to capture the fine scale in the mask (as seen in the gif above). About 9% of the area has a misestimated depth (>0.1 mag), and of course this is spatially correlated.  
87 1 Eli Rykoff
88 9 Eli Rykoff
In terms of getting the bad fraction, things are a bit different.  First, we're dealing with a quantized value when we only have 16 or 64 subpixels, but that's something we can live with.  But even with 16 subpixels we have 5% of the time the masked area is misestimated by more than 5%.  On the other hand, with 64 subpixels only 2% of the pixels are misestimated by > 5% and most of these are only slight outliers.
89 9 Eli Rykoff
90 10 Eli Rykoff
But using nside=4096 with any subpixelization scheme is going to run into the hard limits that it's just not fine enough for the variations in the depth.  How this impacts various science goals is TBD...