SV and DES Data Management¶
DESDM Operations Plan¶
The overall operations plan for DES data processing within DES data management is available in the document database (docDB #6318).
The DESDM team in collaboration with the Fermilab group has prepared a spreadsheet outlining the pieces of the data management
system that are needed for commissioning and during science verification.
This is document docDB #6340 in the document database.
If you are a DES member or participant you will have access to the SV (and commissioning) images through the DES DM system.
First you need to make sure that you are listed in the DES membership database.
If you are not listed, ask your institutional PI to add you.
Secondly you will need a DES DM account. You can register for an account at cosmology.illinois.edu
Eventually we will have a nice user guide providing you will all the information required to access DES images.
For now here is just a brief description
of the basic ideas extracted from an email sent by Don (slightly edited):
DES DM policy is to expose any data to any person with data rights, and to issue personal accounts for each person with data rights. The process starts with the form available at cosmology.illinois.edu. People have to be in the DES membership database. DES DM checks it as part of the account provisioning workflow. DES-DM has the notion of "file pick" by URL, for small accesses, mostly for people who know how to navigate the DESDM file tree and use file names(currently Biases are under path named something like DES/src/20120910/src). _right now it's VAO "pubcookie" authenticated and not scriptable by wget/curl_ This is actively being worked on. They expect to have alternate, scriptable authentication in place for SV. DES-DM is currently commissioning a new feature that will let people query the database (using trivialAccess or any other Oracle client) and produce pathnames to image files satisfying that query. We need to write model queries to overcome SQL impedance. For example: select RELPATH from FILEPATH where ID in (select id from IMAGE where IMAGETYPE = 'raw_bias') This returns the path relative to a root to all the raw_bias files -- these can be fed to FTP or (shortly) curl. And the list WHERE clause can be use to pick from meta-data. The following list is an example of available meta-data: id,run,nite,band,tilename,imagename,ccd,imagetype,project,exposureid,parent id,airmass,exptime,device_id,gaina,rdnoisea,gainb,rdnoiseb,equinox,htmid,cx ,cy,cz,wcsdim,ctype1,cunit1,crval1,crpix1,cd1_1,cd1_2,pv1_1,pv1_2,pv1_3,pv1 _4,pv1_5,pv1_6,pv1_7,pv1_8,pv1_9,pv1_10,ctype2,cunit2,crval2,crpix2,cd2_1,c d2_2,pv2_1,pv2_2,pv2_3,pv2_4,pv2_5,pv2_6,pv2_7,pv2_8,pv2_9,pv2_10,skybrite, skysigma,elliptic,fwhm,scampnum,scampchi,scampflg,pv1_0,pv2_0,naxis1,naxis2 ,nextend,ra,dec,pcount,gcount,bzero,bscale,bunit,ccdname,detsec,ccdsum,data sec,trimsec,ampseca,biasseca,saturata,saturatb,ltm1_1,ltm2_2,ltv1,ltv2,rade sys,wcsaxes,ampsecb,biassecb,zp,zero,sky,seesiga,psfscale,psf_fwhm,psf_beta ,saturate Here is a simple example for a functional query: select RELPATH from FILEPATH where ID in (select id from IMAGE where IMAGETYPE = 'raw_bias') Once you know the pathname, either because you knowledge of the DES-DM directory tree or as a result of a database query one can format the query results into ftp URLs (or soon HTTP URLS) and fetch the data as well.