- Table of contents
- Quick Reduce
- Where we are? (Updated on Aug 2013)
- Querying QR database
Quick Reduce (QR) is a tool designed to run on the mountain integrated with SISPI to monitor the quality of DECam images in real-time.
If you are a DES observer at CTIO or connected to CTIO VPN go to
then login as user:DECamObserver passwd:blanco
then click on "Quick Reduce" -- > "Observing History".
If you are not an observer and want to follow QR in real-time please contact firstname.lastname@example.org and ask for a QR "monitor" account.
Where we are? (Updated on Aug 2013)¶
- QR v0.3 @ CTIO (20 March)
* QR v0.4 @ CTIO - SISPI integration tests (20 April)
* QR v0.5 @ LIneA - Munich Review (15 May)
* QR v0.6 @ CTIO (beg July)
* QR v0.7 @ LineA - (end Ago)
* QR v0.7-3 (Currently Installed at CTIO)
- Basic QR tests
- Actual Performance: process 24 CCDs in about 100s * QR v0.8 @ LIneA (beg Oct) * QR v0.9 @ CTIO (beg Nov)
- QR v1.0 @ CTIO (end Jul)
* QR v1.1 @ CTIO (Aug) Release Notes
* QR v1.2 @ CTIO (?)
- More bug fixing and improvements
- Offline reductions at quick2
The reduction pipeline includes flat and bias corrections and the creation of a SExtractor catalog for each science CCD. These catalogs are used to assess the quality of the data collected during the night by performing several tests:
The following tests are implemented in QR:
- Image check
- Consistency of the FITS header keywords required by QR
- Sky Background
- Median and rms sky background as function of the CCD number
- Map of the spatial distribution of the sky background
- Sky exposure test (bias counts < sky background < maximum counts)
- Number of detected stars as function of the CCD number
- Stray light (variation of the sky background as function of the CCD number)
- Sudden change of sky brightness (Anom-R5.f) (qualitative)
- PSF FWHM
- Median and rms PSF FWHM as function of the CCD number
- Map of the spatial distribution of the PSF FWHM
- PSF FWHM too big compared to site DIMM (Anom-R5.d) (qualitative)
- Sudden change of PSF FWHM (Anom-R5.e)
- PSF ellipticity
- Median and rms of the PSF ellipticity as function of the CCD number
- Ellipticity parameters e1, e2
- Map of the spatial distribution of the PSF ellipticity
- Intensity map of selected stars (brightest non-sturated stars)
- Number of detected objects (greater then the expected number of CRs)
- Depth estimation from galaxy number counts
- Check detection (using FITS viewer tool)
- Overlay the catalog separating stars and galaxies
QR supports two reduction modes, the automatic mode (used during the survey operation) and the manual mode for day time reductions and tests. It is built on the top of the Sicence Portal framework allowing easy integration of science algorithms and tests. The time evolution of the PSF FWHM, PSF elipticity and Sky Background are stored on a database which allows the user to produce a complete view of the Observing history.
Regarding the performance of QR, the CCDs are processed in parallel taking advantage of the multicore machines available at CTIO, currently QR can process 10 CCDs in about 100s. For sampling all the science CCDs after N exposures it is possible to use the "random" option for selecting the CCDs.
The tests results for each exposure are presented in the form of statistics and plots on a web interface and can be shared with the collaboration, this sharing includes a blog like interface for adding comments on each reduced exposure.
The QR results are copied daily to LINEA QR Data Server are can be acessed by the collaboration:
(ask around for usual DES username / password)
QR is available for both survey and community observations.
Bias and flat field correction
Using SExtractor version 2.8.6, the QR default configuration is (will be available soon)
In particular we set the number of pixels above the detection threshold to 20 pixels which avoid spurious detections like bad columns and CRs.
The PSF FWHM is the SExtractor FWHM_WORLD (FWHM in arcsec assuming a gaussian core), the ellipticity is computed as
e = 1 - b/a
and the ellipticity parameters are
e1 = (a-b)/(a+b)*cos(2*theta) e2 = (a-b)/(a+b)*sin(2*theta)
where a = A_IMAGE, b = B_IMAGE and theta = THETA_IMAGE from SExtractor.
The stellar catalog used for PSF analysis has the following cuts:
CLASS_STAR > 0.95 FLAGS < 1 S/N > 200 (*)
(*) this value is being used since 2012-11-17 as we noticed that ellipticity values were contaminated by galaxies in galaxy cluster fields.
Querying QR database¶
Sample query to retrieve information from night, exposure, ccd tables:
SELECT e.expnum AS exposure_id, e.mjd, n.date, e.exptime, e.zd, e.filter, Count(c.fwhm) AS nccds, Avg(fwhm) AS fwhm, Avg(ellipticity) AS ellipticity, Avg(bkg) AS bkg FROM exposure e, ccd c, night n WHERE e.exposure_id = c.exposure_id AND e.night_id = n.night_id AND c.fwhm > 0 AND n.date BETWEEN '2012-11-1' AND '2012-11-21' GROUP BY e.mjd, e.exptime, e.zd, e.filter, e.expnum, n.date ORDER BY e.expnum;
to connect to the DB and execute this query you can use (inside CTIO VPN)
psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c <sql query>
the password is "QRreader", the result is a table like this:
exposure_id | mjd | date | exptime | zd | filter | nccds | fwhm | ellipticity | bkg -------------+--------------+------------+---------+-------+--------+-------+------------------+-------------+--------- 147514 | 56237.017279 | 2012-11-06 | 20 | 55.7 | Y | 10 | 1.40047 | 0.15562 | 30.193 147515 | 56237.032122 | 2012-11-06 | 120 | 0.39 | r | 10 | 2.74341 | 0.46437 | 3.876 147516 | 56237.033867 | 2012-11-06 | 120 | 0.89 | r | 10 | 2.91753 | 0.45101 | 3.924 147517 | 56237.036064 | 2012-11-06 | 120 | 1.64 | r | 10 | 1.62179 | 0.29557 | 3.886 147518 | 56237.037807 | 2012-11-06 | 120 | 2.11 | r | 10 | 1.3237 | 0.16312 | 3.828 147519 | 56237.042727 | 2012-11-06 | 120 | 50.41 | r | 10 | 1.62475 | 0.16936 | 4.218 147520 | 56237.044466 | 2012-11-06 | 120 | 50.41 | r | 10 | 1.80905 | 0.18982 | 4.198 147521 | 56237.046206 | 2012-11-06 | 120 | 50.43 | r | 10 | 1.55274 | 0.11533 | 4.187 147522 | 56237.047948 | 2012-11-06 | 120 | 50.44 | r | 10 | 1.37007 | 0.09587 | 4.267
Night table: store night attributes
psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c "\d night"
Exposure table: exposure attributes
psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c "\d exposure"
CCD table: median and rms parameters for each reduced CCD
psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c "\d ccd"
The Science Portal in a nutshell¶
The Science Portal is an infrastructure project being developed by LINEA for the Dark Energy Survey (DES) collaboration. Science analysis codes prepared by the collaboration (written in C,C++, Fortran, IDL, etc.) can be easily wrapped in the portal. The I/O and configuration are defined in XML files which are used by the Orchestration layer to execute the codes and by the web framework (TurbGears) to automatically render interfaces for editting the configuration and for displaying the results. The python wrappers are the building blocks to create complex workflows in the Portal which are also defined in XML. The Orchestration Layer can interface different job schedulers and execute pipeline blocks in parallel. The pipeline input data can be astronomical images (flat files in FITS format) or catalogs retrieved from a Catalog Database in PostgreSQL.