Project

General

Profile

Quick Reduce

Quick Reduce (QR) is a tool designed to run on the mountain integrated with SISPI to monitor the quality of DECam images in real-time.

QR Start-up guide

If you are a DES observer at CTIO or connected to CTIO VPN go to

http://quick1.ctio.noao.edu:8080/#

then login as user:DECamObserver passwd:blanco

then click on "Quick Reduce" -- > "Observing History".

If you are not an observer and want to follow QR in real-time please contact and ask for a QR "monitor" account.

Where we are? (Updated on Aug 2013)

2012

  • QR v0.3 @ CTIO (20 March)
  • QR v0.4 @ CTIO - SISPI integration tests (20 April)
  • QR v0.5 @ LIneA - Munich Review (15 May)
  • QR v0.6 @ CTIO (beg July)
  • QR v0.7 @ LineA - (end Ago)
  • QR v0.7-3 (Currently Installed at CTIO)
    - Basic QR tests
    - Actual Performance: process 24 CCDs in about 100s
  • QR v0.8 @ LIneA (beg Oct)
  • QR v0.9 @ CTIO (beg Nov)

2013

  • QR v1.0 @ CTIO (end Jul)
  • QR v1.1 @ CTIO (Aug) Release Notes
  • QR v1.2 @ CTIO (?)
    - More bug fixing and improvements
    - Offline reductions at quick2

Description

The reduction pipeline includes flat and bias corrections and the creation of a SExtractor catalog for each science CCD. These catalogs are used to assess the quality of the data collected during the night by performing several tests:

The following tests are implemented in QR:

  • Image check
    - Consistency of the FITS header keywords required by QR
  • Sky Background
    - Median and rms sky background as function of the CCD number
    - Map of the spatial distribution of the sky background
    - Sky exposure test (bias counts < sky background < maximum counts)
    - Number of detected stars as function of the CCD number
    - Stray light (variation of the sky background as function of the CCD number)
    - Sudden change of sky brightness (Anom-R5.f) (qualitative)
  • PSF FWHM
    - Median and rms PSF FWHM as function of the CCD number
    - Map of the spatial distribution of the PSF FWHM
    - PSF FWHM too big compared to site DIMM (Anom-R5.d) (qualitative)
    - Sudden change of PSF FWHM (Anom-R5.e)
  • PSF ellipticity
    - Median and rms of the PSF ellipticity as function of the CCD number
    - Ellipticity parameters e1, e2
    - Map of the spatial distribution of the PSF ellipticity
    - Intensity map of selected stars (brightest non-sturated stars)
  • Number of detected objects (greater then the expected number of CRs)
  • Depth estimation from galaxy number counts
  • Check detection (using FITS viewer tool)
    - Overlay the catalog separating stars and galaxies

QR supports two reduction modes, the automatic mode (used during the survey operation) and the manual mode for day time reductions and tests. It is built on the top of the Sicence Portal framework allowing easy integration of science algorithms and tests. The time evolution of the PSF FWHM, PSF elipticity and Sky Background are stored on a database which allows the user to produce a complete view of the Observing history.

Regarding the performance of QR, the CCDs are processed in parallel taking advantage of the multicore machines available at CTIO, currently QR can process 10 CCDs in about 100s. For sampling all the science CCDs after N exposures it is possible to use the "random" option for selecting the CCDs.

The tests results for each exposure are presented in the form of statistics and plots on a web interface and can be shared with the collaboration, this sharing includes a blog like interface for adding comments on each reduced exposure.

The QR results are copied daily to LINEA QR Data Server are can be acessed by the collaboration:

http://portal.linea.gov.br

(ask around for usual DES username / password)

QR is available for both survey and community observations.

PSF analysis

Image correction

Bias and flat field correction

SExtractor

Using SExtractor version 2.8.6, the QR default configuration is (will be available soon)

In particular we set the number of pixels above the detection threshold to 20 pixels which avoid spurious detections like bad columns and CRs.

The PSF FWHM is the SExtractor FWHM_WORLD (FWHM in arcsec assuming a gaussian core), the ellipticity is computed as

e = 1 - b/a 

and the ellipticity parameters are

e1 = (a-b)/(a+b)*cos(2*theta)
e2 = (a-b)/(a+b)*sin(2*theta)

where a = A_IMAGE, b = B_IMAGE and theta = THETA_IMAGE from SExtractor.

Catalog cuts

The stellar catalog used for PSF analysis has the following cuts:

CLASS_STAR > 0.95
FLAGS < 1
S/N > 200  (*)

(*) this value is being used since 2012-11-17 as we noticed that ellipticity values were contaminated by galaxies in galaxy cluster fields.

Querying QR database

Sample query to retrieve information from night, exposure, ccd tables:

SELECT e.expnum         AS exposure_id,
       e.mjd,
       n.date,
       e.exptime,
       e.zd,
       e.filter,
       Count(c.fwhm)    AS nccds,
       Avg(fwhm)        AS fwhm,
       Avg(ellipticity) AS ellipticity,
       Avg(bkg)         AS bkg
FROM   exposure e,
       ccd c,
       night n
WHERE  e.exposure_id = c.exposure_id
       AND e.night_id = n.night_id
       AND c.fwhm > 0
       AND n.date BETWEEN '2012-11-1' AND '2012-11-21'
GROUP  BY e.mjd,
          e.exptime,
          e.zd,
          e.filter,
          e.expnum,
          n.date 
ORDER BY e.expnum;

to connect to the DB and execute this query you can use (inside CTIO VPN)

psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c <sql query>

the password is "QRreader", the result is a table like this:

 exposure_id |     mjd      |    date    | exptime |  zd   | filter | nccds |       fwhm       | ellipticity |   bkg   
-------------+--------------+------------+---------+-------+--------+-------+------------------+-------------+---------
      147514 | 56237.017279 | 2012-11-06 |      20 |  55.7 | Y      |    10 |          1.40047 |     0.15562 |  30.193
      147515 | 56237.032122 | 2012-11-06 |     120 |  0.39 | r      |    10 |          2.74341 |     0.46437 |   3.876
      147516 | 56237.033867 | 2012-11-06 |     120 |  0.89 | r      |    10 |          2.91753 |     0.45101 |   3.924
      147517 | 56237.036064 | 2012-11-06 |     120 |  1.64 | r      |    10 |          1.62179 |     0.29557 |   3.886
      147518 | 56237.037807 | 2012-11-06 |     120 |  2.11 | r      |    10 |           1.3237 |     0.16312 |   3.828
      147519 | 56237.042727 | 2012-11-06 |     120 | 50.41 | r      |    10 |          1.62475 |     0.16936 |   4.218
      147520 | 56237.044466 | 2012-11-06 |     120 | 50.41 | r      |    10 |          1.80905 |     0.18982 |   4.198
      147521 | 56237.046206 | 2012-11-06 |     120 | 50.43 | r      |    10 |          1.55274 |     0.11533 |   4.187
      147522 | 56237.047948 | 2012-11-06 |     120 | 50.44 | r      |    10 |          1.37007 |     0.09587 |   4.267

Database tables

Night table: store night attributes

psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c "\d night" 

Exposure table: exposure attributes

psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c "\d exposure" 

CCD table: median and rms parameters for each reduced CCD

psql -h server2.ctio.noao.edu -U qr_reader -p 5442 -d desbrdev -c "\d ccd" 

The Science Portal in a nutshell

The Science Portal is an infrastructure project being developed by LINEA for the Dark Energy Survey (DES) collaboration. Science analysis codes prepared by the collaboration (written in C,C++, Fortran, IDL, etc.) can be easily wrapped in the portal. The I/O and configuration are defined in XML files which are used by the Orchestration layer to execute the codes and by the web framework (TurbGears) to automatically render interfaces for editting the configuration and for displaying the results. The python wrappers are the building blocks to create complex workflows in the Portal which are also defined in XML. The Orchestration Layer can interface different job schedulers and execute pipeline blocks in parallel. The pipeline input data can be astronomical images (flat files in FITS format) or catalogs retrieved from a Catalog Database in PostgreSQL.

References

QR Start up guide [01 Nov 2012]
QR Commissioning [22 Oct 2012]
QR Update [18 Sep 2012]
SIWG Demo [21 Jun 2012]
Quick Reduce Review [Munich Meeting]