Project

General

Profile

How to find and how access DECam images at DES DM

All DECam images taken during SV will be sent to DES DM. Images that are not defective are ingested immediately and the (raw) data becomes available soon after. A DESDM account is required in order to access the DES DM system. An account request form is available at the DES DM website.

How to obtain information about the observations

Web Servers

The nightsum daily report, presents
an overview of what happened in the previous night.

There are four web servers at CTIO that provide various types of information
about the files. Three of these web servers are accessible via FNAL Mirrors:

The first one (Exposure Viewer) may be the most useful, as it contains file
numbers and a brief description of each file. Note that if you click the
'Edit' button, you can modify the SQL query. The output may also be exported
as csv, etc.

These DB may also be queried from the command line. To do this you can use the psql program.

  1. You will need to log into an FNAL computer
  2. Set the following environment variables:
  1. setenv PGHOST des20.fnal.gov
  2. setenv PGPORT 5443
  3. setenv PGDATABASE decam_prd
  4. setenv PGUSER decam_reader
  5. setenv PGPASSWORD reader
  1. A sample command to make a csv file is:
    psql -A -F"," --pset footer=off -f dbquery-part1.sql > db-part1.csv &
  2. A very extensive query is in dbquery-part1.sql and dbquery-part2.sql

The four web servers accessible through CTIO are:

Access to the CTIO web servers requires one to either be at CTIO or have an
FNAL IP address. Here are instructions on how to Setup A FNAL IP Address.

Note that the links to the png files of the images in the Exposure Viewer
point to CTIO so those will not be visible from offsite unless you use the
CTIO versions. The logbook cannot use the FNAL mirror because it needs to
access the database in read/write mode even if you are only viewing pages.

The Daily Report:
docdb-6653
contains useful information up to 9-25-2012. It has now been replaced by the
nightsum daily report mentioned
above.

How to obtain the files

DES DM Help

Database Access

EasyAccess

The standard tool provided by DES DM to find images in the DES DM database is
EasyAccess. This file contains a few hints and additional information
about EasyAccess.

Additional instructions and examples are found in:
https://opensource.ncsa.illinois.edu/confluence/display/DESDM/EasyAccess

  • Once you have EasyAccess installed you can start it from the command line:
    easyaccess 
    

or for help,

easyaccess -h

The first time you start the program a read-only-by-user credential file will be created in your home directory after prompting for username and password
This is a one-time occurrence.
It's not good practice to enter your credentials on the command line every time you run EasyAccess.

  • EasyAccess is interactive and offers you a prompt ("DESDM ~>") to enter commands.
    Try "help" to see a list of commands.
Example: (taken from trivialaccess description)
A sample query to retrieve raw image files from DES DM.
select ('wget ' ||  'ftp://desar.cosmology.illinois.edu/DESFiles/desardata/' || filepath.path || '.fz') \
as path  from exposure, FILEPATH where exposure.id = filepath.id and obstype = 'dome flat' \
and exposure.band = 'i'  and exposure.project = 'DTS'  and  nite=20120824 

This returns the full path to dome flats (raw image files) from night 2012-08-24 and filter i.
Useful keywords for the select command include:
nite - night of the exposure
band - filter
obstype - (SISPI) exposure type (OBSTYP keyword)
expnum - exposure numper
  • The select command will (can) return a list of filenames that you can use in a script with your
    favorite tool like wget to recursively grab the files.

Easyaccess tutorial: http://deslogin.cosmology.illinois.edu/~mcarras2/DESDM.html#/

Attached to this page (see bottom) is a tar-ball containing a simple script which will download the exposure catalogs from First Cut processing at NCSA, read from an user-defined exposure list. It requires easyAccess installed and HEATOOLS if you want to automatically merge the catalogs as well.

pythonAccess

Easyaccess can be imported as a python module as well, check the online tutorial: http://deslogin.cosmology.illinois.edu/~mcarras2/DESDM.html#/

In addition, Erin Sheldon has developed a python package desdb that accesses the DES
DM database directly from python without requiring jython.

https://github.com/esheldon/desdb.

This package also has scripts to automate the download of reduced and coadd
image data.

The get-raw-images
script allows you to select filter, exposure type, exposure number range,
night etc for raw, unreduced data from the command line.

(Early) Web Interface

A very simple web-based query interface is available, An NVO account is required:
  • Query page
  • Slightly outdated information about the file tree can be found here
    The short course is that below each release there are files, sorted by class:
    cal/             # calibrations
    coadd/           # tiles and data related to coaddition
    diff/            # SNE files.
    red/             # reduced images, and remaps
    src/             # input images
    wl/              # weak lensing
    

A Portal Interface is under development. There is a form where you can specify the files you want by position band and class within a release. The portal system bundles the files in a tarball, and sends email with a link allowing you to fetch it. [Not yet operational]

Image and Catalog Flat Files

How to find a DES Image using a database query

If you want to find files DES DM supports a query mechanism that is documented
here

DES File Servers

The top level directory, for browsing or downloads, is here
https://desar2.cosmology.illinois.edu/DESFiles/desardata

Once you have located an image file or directory are interested in you can
browse it or copy it using wget. For example

url=https://desar2.cosmology.illinois.edu/DESFiles/desardata/OPS/coadd/20130227000003_DES0102-4914/coadd/DES0102-4914_r.fits.fz
wget --no-check-certificate $url

If you want an entire exposure or run, it is much more efficient to sync it
all at once. For an exposure
run=20130402134538_20130326
expname=DECam_00190760
url=https://desar2.cosmology.illinois.edu/DESFiles/desardata/OPS/red/${run}/red/${expname}/
cut_dirs=7
wget                        \
    --no-check-certificate  \
    -c                      \
    -nv                     \
    --mirror                \
    -nH                     \
    --cut-dirs=$cut_dirs    \
    --no-parent             \
    --tries 50              \
    "$url" 

For an entire run, cut off the exposure name and use cut-dirs=6

run=20130402134538_20130326
url=https://desar2.cosmology.illinois.edu/DESFiles/desardata/OPS/red/${run}/red/
cut_dirs=6
wget                        \
    --no-check-certificate  \
    -c                      \
    -nv                     \
    --mirror                \
    -nH                     \
    --cut-dirs=$cut_dirs    \
    --no-parent             \
    --tries 50              \
    "$url" 

The file server uses the same username and password as the database. The
easiest way to send your password is using a ~/.netrc file, which wget will
automatically check

machine desar2.cosmology.illinois.edu login {your_login} password {your_pass}

Replace {your_login} with your actual login, etc.

Directory Structure at NCSA

The reduced data will appear under OPS.

    /OPS/red   - reduced single epoch images and catalogs
    /OPS/coadd - coadd images and catalogs

The raw data appear under DTS.

You can find things manually by browsing the above, though the other methods
described on this pagewill allow you to find data more systematically.

Useful Information