Running the DES precal Pipeline

Brian and Marko's Introduction

This note is to announce the second release of
'unit running' stand-alone DESDM software and associated
'wrappers'. It is intended to put the
SV team on a path where they can run pieces of the
software on their own home or mountain-top Linux box
to test early DECAM data using the
same codes that will be used for production processing.

One needs to have the 6.3.2 DES DM stack (or later) and the DES_HOME
codes installed on your workstation before you can try this.

This package allows one to:
  • split exposures into 62 CCDs (or, in test mode, just read off the first N HDUs).
  • read the CCDNUMs 63-70 (alignment/focus)
  • combine N bias exposures into biascor frames.
  • combine N flat exposures into flatcor frames.

Remember to run the setup_desdm script (in $DES_BASE/setup_desdm if you followed the stack installation instructions)


#!/bin/bash -x

#Aug 23, 2012
# This note contains instructions for setting up and
# runing v2 of the 'precal unit running' pipeline.
# The main change is that mkflatcor is now supported
# in addition to basic split-and-crosstalk and mkbiascor.
# mkflatcor does a clipped average of N input flat exposures
# (after splitting and crosstalk correcting them).
# mkflatcor outputs a set of decam-flat-$filter_$CCD_dflatcor.fits 
# images to be used  later by imcorrect 
# (along with decam-bias-$CCD_dbiascor.fits biases) to 
# flatten science exposures.
# Follow the instructions below, 
# note the 'side step' of compiling your own version of DECam_crosstalk.c
# until we get the corrected version checked in to SVN trunk.
# Create yourself a test directory
mkdir test
cd test

#set links or copy executables to DES_HOME, and stack dirs
# DES_HOME/bin has codes like mkbiascor,DECam_crosstalk
# stack/bin has codes like sextrator, scamp

ln -s $DES_BASE/{DES_HOME,stack} .

# Make sure the right python version is in your PATH and PYTHONPATH is set correctly

# We suggest you get the 'modified for 70 chips' version of DECam_crosstalk:
# put it in your  $DES_BASE/desdm/devel/ImageProc/trunk/src area
# edit the Makefile to only recompile DECam_crosstalk.c
# or recompile the whole set  (make install)

# cd into the Wrappers, IntegrationUtils dirs and do  'svn update'
svn co
svn co

# get the new test data, note that this testdata file is 2.2GB!

# create a directory and unpack the image data in that directory
mkdir runindir
cd runindir
tar xzf ../precal_test_dataB.tgz

# That's it for one-time setup. You don't have to repeat these steps again.

# Now each time you want to run scripts
# cd runindir and do the following to setup (should return null if ok):
source $DES_BASE/desdm/devel/Wrappers/trunk/utils/  $DES_BASE

# try the test program with the testdata -t
# this will run and state whether it succeeds or not if it does succeed it cleans up after itself.

# if you run -t -k
# the '-k' flag say 'keep the outputs around after running the test'.

# They are in a subdirectory called temp_test_bias and temp_test_flat
# If that returns ok, then you can try other things yourself
# you will need to have the  special version of DECam_crosstalk (see instructions above)
# in order to get this test to work on just 2 CCDs.

# We recommend you run -t -k and then
# you will have some sample outputs in tmp_test_bias and tmp_test_flat
# which you can edit and try out your own things.

# for instance, you will have in the testdata dir a 
# raw exposure flat called DECam_t183_01000000.fits.fz
# to run split-and-crosstalk on it, do the following:
cd tmp_test_flat
cp wcl_0.wcl wcl_test10.wcl
# and then edit wcl_0.wcl changing at the bottom:  maxhdunum = 3 --> maxhdunum = 10
# also change range = 0-1 to range = 0 in the 'loop' section.
# then execute this wrapper with this command: --config wcl_test10.wcl

# this will read off the first 9 (=10-1) HDUs of the flat exposure 01000000
# and leave you with 10 files of the form DECam_t183_01000000_$CCD.fits
# Where CCD is 13,19,20,25,26,27,32,33,34
# you can change the maxhdunum = 10 to other numbers (try 71, try 63)
# and you can change the range from range=0 to range=1 to 
# do the same to the other flat exposure...

# a 'full test' of the precal script includes
# running it on 62 ccds.  You will need to supply a DECam_xtalk.dat file
# and 62 pupil and 62 bpm images  before this will work.
# The cross talk file is available from this page. You can retrieve a set of 62
# pupil and bpm image using the command:
# (Note that this is a 0.8 GB file)
# To use the checked-in wcl, the following should all be in the same directory 
# where the precal is going to be done:

  * the bias and pupil images from precal_pupil_bpm_imagesB.tgz,
  * the xtalk file from this page,
  * the raw bias exposures(DECam_t183_01000050.fits.fz
    and DECam_t183_01000051.fits.fz) and flat exposures
    (DECam_t183_01000000.fits.fz and
    DECam_t183_01000001.fits.fz) from

# The command for running all 62 chip is -b 'DECam_t183_010000${loop.exposure}.fits.fz' -r '50-51' -f 'DECam_t183_0100000${loop.exposure}.fits.fz' -s '0-1'
# This takes about 30 minutes