Project

General

Profile

SV and DES Data Management

DESDM Operations Plan

The overall operations plan for DES data processing within DES data management is available in the document database (docDB #6318).

Deliverables

The DESDM team in collaboration with the Fermilab group has prepared a spreadsheet outlining the pieces of the data management
system that are needed for commissioning and during science verification.
This is document docDB #6340 in the document database.

Data Access

If you are a DES member or participant you will have access to the SV (and commissioning) images through the DES DM system.

First you need to make sure that you are listed in the DES membership database.
If you are not listed, ask your institutional PI to add you.

Secondly you will need a DES DM account. You can register for an account at cosmology.illinois.edu

Eventually we will have a nice user guide providing you will all the information required to access DES images.
For now here is just a brief description
of the basic ideas extracted from an email sent by Don (slightly edited):

DES DM policy is to expose any data to any person with data rights, and to issue personal accounts for each person with 
data rights. The process starts with the form available at cosmology.illinois.edu.  People have to be in the 
DES membership database. DES DM checks it as part of the account provisioning workflow.

DES-DM has the notion of "file pick"  by URL, for small accesses, mostly for people who know how to 
navigate the DESDM file tree and use file names(currently Biases are under path named  something 
like DES/src/20120910/src).

_right now it's VAO "pubcookie" authenticated and not scriptable by wget/curl_

This is actively being worked on. They expect to have alternate, scriptable authentication in place for SV.

DES-DM is currently commissioning a new feature that will let people query the database (using trivialAccess or 
any other Oracle client) and produce pathnames to image files satisfying that query.
We need to write model queries to overcome SQL impedance.  For example:
 select RELPATH from FILEPATH where ID in (select id  from IMAGE where IMAGETYPE = 'raw_bias')
This returns the path relative to a root to all the raw_bias files -- these can be fed to FTP or (shortly) curl.  
And the list WHERE clause can be use to pick from meta-data.

The following list is an example of available meta-data:
id,run,nite,band,tilename,imagename,ccd,imagetype,project,exposureid,parent
id,airmass,exptime,device_id,gaina,rdnoisea,gainb,rdnoiseb,equinox,htmid,cx
,cy,cz,wcsdim,ctype1,cunit1,crval1,crpix1,cd1_1,cd1_2,pv1_1,pv1_2,pv1_3,pv1
_4,pv1_5,pv1_6,pv1_7,pv1_8,pv1_9,pv1_10,ctype2,cunit2,crval2,crpix2,cd2_1,c
d2_2,pv2_1,pv2_2,pv2_3,pv2_4,pv2_5,pv2_6,pv2_7,pv2_8,pv2_9,pv2_10,skybrite,
skysigma,elliptic,fwhm,scampnum,scampchi,scampflg,pv1_0,pv2_0,naxis1,naxis2
,nextend,ra,dec,pcount,gcount,bzero,bscale,bunit,ccdname,detsec,ccdsum,data
sec,trimsec,ampseca,biasseca,saturata,saturatb,ltm1_1,ltm2_2,ltv1,ltv2,rade
sys,wcsaxes,ampsecb,biassecb,zp,zero,sky,seesiga,psfscale,psf_fwhm,psf_beta
,saturate

Here is a simple example for a functional query:
   select RELPATH from FILEPATH where ID in (select id  from IMAGE where IMAGETYPE = 'raw_bias')

Once you know the pathname, either because you knowledge of the DES-DM directory tree or as a result of a
database query one can format the query results into ftp URLs (or soon HTTP URLS) and fetch the data as well.