Project

General

Profile

Balrog Run 1

Balrog Run 1 is complete, including magnification test for LSS:

running on 1017 Y3 tiles taken from the list in YANNY.ALLBAL@DESSCI.

For stacked Balrog outputs, look here

Magnification test details at:

https://data.darkenergysurvey.org/fnalmisc/balrog/run1/

Summary: 4.4M deep field galaxies injected into ~1000 tiles (10% subsample) two ways:
1. with no magnification factor (1.00)' and run through SOF
2. with 1.02 (2%) magnification factor and rerun through SOF (exactly the same galaxies).

Grab matched file between truth_table <---> magnify=1.00 SOF <---> magnify=1.02 SOF mags and sizes in one place:

https://data.darkenergysurvey.org/fnalmisc/balrog/run1/balrog_sof_magnification_test.fits

Metacal runs packaged into .h5 format are available at the data.darkenergysurvey.org file:

http://data.darkenergysurvey.org/fnalmisc/balrog/run1/

wget --http-password=XXX --http-user=XXXX https://data.darkenergysurvey.org/fnalmisc/balrog/run1/balrog_mcal_stack-y3v02-0-griz-noNB-mcal.h5

balrog_mcal_stack-y3v02-0-griz-mcal.h5

Tiles in the Balrog Run 1 subset have inbalrog1==1 in YANNY.ALLBAL@DESSCI.
Note that there are a few duplicate tiles in ALLBAL, as they appear in both the 'random' tile
tilegroup and again in the y1svblock tilegroup (etc).

The balrog outputs are in fits format on Fermilab computers in the directory:
des41.fnal.gov:/data/des41.b/data/Balrog/prod020419/DES<TILENAME>/
Where <TILENAME> is one of 1017 completed Y3 tiles.

Approximately 11,000 deep field galaxies and 2,500 stars are injected into
each of these tiles, and then the tile with the extra objects are
reprocessed and all objects redetected and remeasured.

We have begun to do matches of the Balrog outputs to the injected input object
deep field catalogs and make these available as FITS files and as DESSCI database
tables.

To make access easier for initial use of the Balrog outputs (including for
SOMPZ, LSS and cluster analysis)
a match has been performed between the input galaxies injected and the resulting
detected SOF measurements of these objects and been placed into matched
database table(s):

The fits files are available in the directory and the note here:

https://data.darkenergysurvey.org/fnalmisc/balrog/run1/AAAREADME.DAT
(use the usual default des authentication credential)

Where 5 .fits catalogs are available:

balrog_matched_catalog.fits: Matches between 4.5M true and
measured balrog objects 13.6GB

balrog_matched_catalog_flatten.fit: subset version of the same file
(some columns left out), with reduced columns
and arrays flattened out. Smaller (3.5GB)
I.E. CM_MAG_DEREDDEN4 --> (CM_MAG_DEREDDEN_[griz]).

balrog_detection_catalog.fits: Truth cat properties and ID of
11.4M injected balrog objects (of which 1/3 are
detected and are in the Matched catalog, above): 1.3GB

balrog_detection_catalog_flatten.fit: version of the same file,
with reduced columns and arrays flattened out.

stacked_sof_truth_match.fits: Subset of balrog_matched_catalog.fit+detection_catalog.fits, with column names flattened: 5GB (old)

Three of these files are also available in the DESSCI database:

YANNY.BALROG_MATCHED_CATALOG_FLATTEN: SOF<--> TRUTH matchup for 4M detected matches

YANNY.BALROG_DETECTION_CATALOG_FLATTEN: TRUTH (COSMOS_V3 and SN COADD_TRUTH Wide deep field objects) used in the injection of Galaxies into the 1,017 Balrog tiles (11.4M objects). Note that this catalog ha entries even for objects which are not detected in the matched catalog above (usually because they are too faint).

One does a join on the 'BAL_ID' field between the MATCHED and the DETECTION catalog if you wish to link them
together. If you to join to the COSMOS_V3 or COADD_TRUTH tables you link between 'true_id' in the Balrog table and 'COADD_OBJECT_ID' in the DEEP Field tables. You may also need to use 'true_tilename' to figure out which DEEP Field table to link to.

YANNY.STACKED_SOF_TRUTH_MATCH: This is an older version of YANNY.BALROG_MATCHED_CATALOG_FLATTEN which has less fields, but also has all the detection (without matches) present. We recommend using YANNY.BALROG_MATCHED_CATALOG_FLATTEN and YANNY.BALROG_DETECTION_CATALOG_FLATTEN for now, but this older catalog
will be kept around a while.

Note that the Star (delta function injections) have not yet been made into matched catalogs. This is coming soon.

YANNY.STACKED_SOF_TRUTH_MATCH@DESSCI.

DESCRIBE_TABLE YANNY.STACKED_SOF_TRUTH_MATCH

gives all the columns available.

Only the diagonal elements of the MEAS_CM_FLUX_COV are given, but all the off diagonal elements are -9999 in the
Balrog outputs files, so there's no further info available there.

This table has 11.4M rows, one for each injected galaxy. About 4.5M of these objects
were detected -- the rest were too faint to be detected. If an object is not
detected the meas_flags column for that object and other measured quantities are
-9999. In this way completeness as a function of magnitude and color and size
may be determined.

For each object injected parameters (true_ra,true_dec,true_cm_flux_[griz], true_cm_T, true_cm_T_err, true_cm_fracdev, true_cm_g_{12},
true_cm_flux_deredden_[griz]) are available. Injected objects are identified by their (coadd_object_id,true_tilename) pair
which gives an id lookup into DESDM's Y3A2_MISC deep field tables (either the COSMOS_V3 table for cosmos deep field
objects or the SN deep tile COADD_TRUTH table, one can determine which by looking at the true_tilename parameter here if
coadd_object_id is not unique). Input quantities true_psf_mag_[griz], true_flags, true_cm_flags and delta_cm_mag_deredden_[griz] values are also available.

When there is a match to a detected object on the Balrog injected tiles, the measure SOF quantities have matching values given
by the meas_ra, meas_dec, meas_cm_flux_[griz], meas_cm_flux_cov_[gg,rr,ii,zz], meas_cm_T, meas_cm_T_err, meas_cm_fracdev,
meas_cm_fracdev_err, meas_cm_flux_deredden_[griz], meas_cm_mag_deredden_[griz], meas_tilename (which Y3 tile was the object injected
into), meas_flags, meas_cm_flags quantities match those with true_ prefix (above) and may be directly compared in most cases.
Note that one should compare the dereddened fluxes with dereddened fluxes.

This DESSCI DB table is indexed on coadd_object_id, true_ra, true_dec, meas_ra, meas_dec, true_tilename, meas_tilename, meas_flags.
Note that there's no true_cm_mag_deredden_[griz] but only delta_cm_mag_deredden = meas_cm_mag_deredden - true_cm_mag_deredden for each of g,r,i and z.

While this table includes SOF measured mags, MOF, BFD and Gaussian Aperture mag matches can also be made available.

This table only involves galaxies, but a similar matched table for the 2,500 stars/tile that were injected can also be made available.

The table is also available as one large Binary FITS table from:

http://data.darkenergysurvey.org/fnalmisc/balrog/run1/
Where you may download with:

wget https://cdcvs.fnal.gov/redmine/projects/des/wiki/Balrog_Run/stacked_sof_truth_match.fits

using the usual DES auth. credentials.

Sample Database query, to grab the input (true) and the output (meas) fluxes and S/N (flux/sqrt(flux_cov)) for each band griz for objects on one tile:

  select meas_cm_mag_deredden_g-delta_cm_mag_deredden_g as true_g0,delta_cm_mag_deredden_g,
  meas_cm_flux_g/sqrt(meas_cm_flux_cov_gg) as gs2n,
  meas_cm_mag_deredden_r-delta_cm_mag_deredden_r as true_r0,
  delta_cm_mag_deredden_r,meas_cm_flux_r/sqrt(meas_cm_flux_cov_rr) as rs2n,
  meas_cm_mag_deredden_i-delta_cm_mag_deredden_i as true_i0,delta_cm_mag_deredden_i,
  meas_cm_flux_i/sqrt(meas_cm_flux_cov_ii) as is2n,
  meas_cm_mag_deredden_z-delta_cm_mag_deredden_z as true_z0,
  delta_cm_mag_deredden_z,meas_cm_flux_z/sqrt(meas_cm_flux_cov_zz) as zs2n,
  meas_flags from yanny.stacked_sof_truth_match where meas_tilename = 'DES2253-4623' and meas_cm_flags=0; >d2253.fits

One may then run a general matplotlib/python plotting script to plot mag vs. delta mag and put some sort of error bars on the plot:

Here's a sample python/matplotlib script to make the above plot:

#!/usr/bin/env/python
#from esutil import htm
from astropy.table import Table
import numpy
import matplotlib.pyplot as plt
import sys
import string
#from astropy.io import fits

infile=sys.argv[1]
intile=sys.argv[2]
xlo=float(sys.argv[3])
xhi=float(sys.argv[4])
ylo=float(sys.argv[5])
yhi=float(sys.argv[6])
suffix=sys.argv[7]

tg=Table.read(infile)
t=tg[numpy.where(tg['MEAS_FLAGS']==0)]
print "len: ", len(tg), " with flags==0: ",len(t)

x1=t['TRUE_G0']
x2=t['TRUE_R0']
x3=t['TRUE_I0']
x4=t['TRUE_Z0']

y1=t['DELTA_CM_MAG_DEREDDEN_G']
y2=t['DELTA_CM_MAG_DEREDDEN_R']
y3=t['DELTA_CM_MAG_DEREDDEN_I']
y4=t['DELTA_CM_MAG_DEREDDEN_Z']

#s2n1=t['GS2N']
#s2n2=t['RS2N']
#s2n3=t['IS2N']
#s2n4=t['ZS2N']

x1label='true_cm_mag_g0'
x2label='true_cm_mag_r0'
x3label='true_cm_mag_i0'
x4label='true_cm_mag_z0'

y1label='(meas-true)(cm_mag_g0)'
y2label='(meas-true)(cm_mag_r0)'
y3label='(meas-true)(cm_mag_i0)'
y4label='(meas-true)(cm_mag_z0)'

fig,((ax1,ax2),(ax3,ax4)) = plt.subplots(2,2,sharex='col',sharey='row')

ax1.set_xlim([xlo,xhi])
ax1.set_ylim([ylo,yhi])
ax2.set_xlim([xlo,xhi])
ax2.set_ylim([ylo,yhi])
ax3.set_xlim([xlo,xhi])
ax3.set_ylim([ylo,yhi])
ax4.set_xlim([xlo,xhi])
ax4.set_ylim([ylo,yhi])

ax1.set(xlabel=x1label,ylabel=y1label)
ax1.scatter(x1,y1,s=4,c='blue',marker='.')

ax2.set(xlabel=x2label,ylabel=y2label)
ax2.scatter(x2,y2,s=4,c='green',marker='.')

ax3.set(xlabel=x3label,ylabel=y3label)
ax3.scatter(x3,y3,s=4,c='red',marker='.')

ax4.set(xlabel=x4label,ylabel=y4label)
ax4.scatter(x4,y4,s=4,c='orange',marker='.')

ax1.axhline(y=0,color='black',linestyle='-')
ax2.axhline(y=0,color='black',linestyle='-')
ax3.axhline(y=0,color='black',linestyle='-')
ax4.axhline(y=0,color='black',linestyle='-')

for m in (16,16.5,17,17.5,18,18.5,19,19.5,20,20.5,21,21.5,22,22.5,23,23.5,24):
  hi=m+0.5
  midpt=(m+hi)/2.0

  subg=t[numpy.where(t['TRUE_G0']>m)]
  subg2=subg[numpy.where(subg['TRUE_G0']<hi)]
  ptoffsetg=numpy.median(subg2['DELTA_CM_MAG_DEREDDEN_G'])
  ptheightg=1.414*1.0857/numpy.mean(subg2['GS2N'])
  print midpt,ptheightg,ptoffsetg
  ax1.plot(midpt,ptoffsetg,color='magenta',markersize=4,marker='o')
  ax1.errorbar(midpt,ptoffsetg,yerr=ptheightg,c='magenta',linewidth=2.0)

  subr=t[numpy.where(t['TRUE_R0']>m)]
  subr2=subg[numpy.where(subg['TRUE_R0']<hi)]
  ptoffsetr=numpy.median(subg2['DELTA_CM_MAG_DEREDDEN_R'])
  ptheightr=1.414*1.0857/numpy.median(subr2['RS2N'])
  ax2.plot(midpt,ptoffsetr,color='magenta',markersize=4,marker='o')
  ax2.errorbar(midpt,ptoffsetr,yerr=ptheightr,c='magenta',linewidth=2.0)

  subi=t[numpy.where(t['TRUE_I0']>m)]
  subi2=subg[numpy.where(subg['TRUE_I0']<hi)]
  ptoffseti=numpy.median(subg2['DELTA_CM_MAG_DEREDDEN_I'])
  ptheighti=1.414*1.0857/numpy.median(subi2['IS2N'])
  ax3.plot(midpt,ptoffseti,color='magenta',markersize=4,marker='o')
  ax3.errorbar(midpt,ptoffseti,yerr=ptheighti,c='magenta',linewidth=2.0)

  subz=t[numpy.where(t['TRUE_Z0']>m)]
  subz2=subg[numpy.where(subg['TRUE_Z0']<hi)]
  ptoffsetz=numpy.median(subg2['DELTA_CM_MAG_DEREDDEN_Z'])
  ptheightz=1.414*1.0857/numpy.median(subz2['ZS2N'])
  ax4.plot(midpt,ptoffsetz,color='magenta',markersize=4,marker='o')
  ax4.errorbar(midpt,ptoffsetz,yerr=ptheightz,c='magenta',linewidth=2.0)

ax1.set_title(intile)
ax2.set_title('N='+str(len(t)))
fig.savefig(intile+suffix+'.png')

Running this script aplot.py like so:

python aplot.py /data/des80.a/data/yanny/d2253.fits DES2253-4653 14 27 -0.2 0.2 deltamag

To generate:

DES2253-4653deltamag.png

Please try this out and let us know if these Balrog Run 1 outputs can help your Y3 science!

Balrog matches to deep field and photo-z's available on this page