Project

General

Profile

Running the standalone pipeline with python

General information

Version 7 of the standalone data reduction pipeline is installed on the computer system at CTIO. It can be run in two different ways:
  1. "Out of the box" by manipulating .wcl files manually.
  2. Eric Suchyta has written python wrappers around the .wcl environment which does all the .wcl work for you under the hood. It supports changing parameters in the pipelines from the command line or from simple input configuration files. Using this version is described in more detail below.
The "home" directory for the files is /home2/sispiuser/detrending/submit_by_script. If you plan on using the python wrappers, it is suggested you copy the entire "home" area to a working area of your own and then make changes accordingly. Within the "home" directory, you'll find some subdirectories:
  • python -- contains the python wrappers. The files you will run are in here.
  • param -- contains example input parameter files. In short, the files contain a different command line argument on each line.
  • list -- contains example list files which are used for the precal pipeline.
  • astron_config -- contains example configuration files for astronomy software such as sextractor and scamp.
  • custom_wcl -- contains .wcl files which the wrappers will edit. You should not need to edit these directly.
  • old -- contains old version of various things. You will not need anything here. It will probably be deleted soon.
There are 10 different python script files in the python directory. These can be thought of as five pairs of files:
  • run_mkbiascor.py, run_mkbiascor_script.py -- runs DECam_crosstalk and mkbiascor, i.e. combines individual bias frames into a master bias.
  • run_mkflatcor.py, run_mkflatcor_script.py -- runs DECam_crosstalk and mkflatcor, i.e. combines individual flat frames into a master flat.
  • run_firstcut.py, run_firstcut_script.py -- runs firstcut pipeline (DECam_crosstalk --> imcorrect --> sextractor --> scamp --> sextractor), i.e. reduces an image, updates astrometry, makes a catalog of sources.
  • run_finalcut.py, run_finalcut_script.py -- runs finalcut pipeline, in addition to firstcut this attempts to mask bright stars and models the PSF for sextractor.
  • run_mycut.py, run_mycut_script.py -- a "combination" of firstcut and finalcut I made up which doe not do the mask and does not do the final catalog. I am able to turn parts of different pipelines "on" and "off".

The first file in each of the pairs will run with command line arguments, while the second file in each of the pairs will run with a configuration file as its argument. (The second in the pair calls the first in the pair and converts the file to command line arguments.)

Command line arguments

Each of the python wrappers take some mandatory parameters as well as optional command line arguments. Any optional arguments not specified take on default values. The mandatory parameters will be noted below in the description of each wrapper. Here a few general comments are made about the optional parameters.

The optional command line arguments use the same syntax as the individual components of the astronomy codes described on the pipeline software components page here:
[[https://desdb.cosmology.illinois.edu/confluence/display/DESAST/Pipeline+Software+Components]]
(You will need a DESDM username and password to access the page). Most arguments are in the form of -PARAMETER VALUE pairs, while some have the form -FLAG. There is one parameter invented for the wrappers called -ccdlist LIST. This specifies which CCDs you would like to use. The default is -ccdlist 1-62 (i.e. all the science CCDs), but if you only want to look at certain CCDs this can be done, e.g. -ccdlist 25,26.

Command line arguments are specified for any different portion of a pipeline at the same time. For example run_mkflatcor.py calls DECam_crosstalk and mkflatcor. You would specify both -overscansample -1 and -avsigclip 5 if you wanted the DECam_crosstalk to use medians in the overscan correction and mkflatcor to reject 5 sigma outliers.

For the firscut, finalcut, and mycut, you can specify the config/parameter files for the astronomy codes such as sextractor and scamp. They are named based on where in the pipeline they occur. For example -sexforscamp_config file specifies the config file for the sextractor run which occurs before scamp. -sexforscamp_param is the output parameter file. See the list below for more details about the different pipelines:
1) firstcut:
  • -sexforscamp_config <FILE>
  • -sexforscamp_param <FILE>
  • -scamp <FILE>
  • -sexforcat_param <FILE>
  • -sexforcat_config <FILE>
2) finalcut:
  • -sexforscamp_config <FILE>
  • -sexforscamp_param <FILE>
  • -scamp <FILE>
  • -sexformask_config <FILE>
  • -sexformask_param <FILE>
  • -sexforpsfex_config <FILE>
  • -sexforpsfex_param <FILE>
  • -psfex <FILE>
  • -sexforcat_param <FILE>
  • -sexforcat_config <FILE>
3) mycut:
  • -sexforscamp_config <FILE>
  • -sexforscamp_param <FILE>
  • -scamp <FILE>
  • -sexforpsfex_config <FILE>
  • -sexforpsfex_param <FILE>
  • -psfex <FILE>

Deprecated parameters are not supported. For example, overscan corrections are only allowed to be done in DECam_crosstalk. So the overscan options to any other commands are not available. At the moment, certain static calibration frames are assumed to be in a particular directory, and they are just null corrections. As bpm, crosstalk matrix, pupil, etc. become available this will be updated. (The null corrections are the ones distributed with standalone pipeline.)

Configuration files

Each line of the configuration file is one command line argument, including if applicable both the -PARAMETER VALUE. Here is an example of the contents of a configuration file for firstcut (in the param subdirectory at firstcut.param):

00136100
/n/des/des/images/first_light/eric/20120924_precal_v7_fs/
z
/n/des/des/images/first_light/eric/20120924_firstcut_tmp

-ccdlist 1
-sexforscamp_config ../astro_config/sexforscamp.config
-sexforscamp_param ../astro_config/sexforscamp.param
-scamp ../astro_config/scamp.config
-sexforcat_config ../astro_config/sexforcat_nopsf.config
-sexforcat_param ../astro_config/sexforcat_nopsf.param

Calls/Examples

run_mkbiascor.py <input_list> <output_directory> (-OPTIONAL)
./run_mkbiascor.py ../list/bias.list /home2/sispiuser/detrending/suchyta_detrend/precal_test -overscansample -1 -avsigclip 5
./run_mkbiascor_script.py ../param/bias.param

run_mkflatcor.py <input_list> <filter_band> <output_directory> (-OPTIONAL)
./run_mkflatcor.py ../list/flat_z.list /home2/sispiuser/detrending/suchyta_detrend/precal_test -overscansample -1 -avsigclip 5
./run_mkflatcor_script.py ../param/flat.param

run_firstcut.py <image_number> <flat_bias_dir> <filter_band> <output_directory> (-OPTIONAL)
./run_firstcut.py 00136100 /your/dir/precal z /our/dir/firstcut -overscansample -1 -sexforscamp ../astro_config/sexforscamp.config
./run_firstcut.py ../param/firstcut.param

run_finalcut.py <flat_bias_dir> <filter_band> <output_directory> (-OPTIONAL)

run_mycut.py <flat_bias_dir> <filter_band> <output_directory> (-OPTIONAL)

Other Details

Because of the way the standalone pipeline is written, all of these must be run under the bash shell. I have added the following to the .bashrc for my username.

dir=$(pwd)
cd /usr/remote/data1/sispiuser/DESDM
source DES_HOME/bin/setup_runB.sh .
export DES_HOME_DATA=/home3/data_local/images/fits/2012B-9999
export DES_HOME_SC=/home2/sispiuser/detrending/runindir_v6/precal_test_dataC/cal
cd $dir

This sets up some necessary environment variables, as well as some which make things easier in the python wrappers. The images reduced at CTIO are assumed to live in the directory:
/home3/data_local/images/fits/2012B-9999. /home2/sispiuser/detrending/runindir_v6/precal_test_dataC/cal is where the static calibration frames live. For now these are the null ones I mentioned above. By editing these two lines of the .bashrc file:

export DES_HOME_DATA=/home3/data_local/images/fits/2012B-9999
export DES_HOME_SC=/home2/sispiuser/detrending/runindir_v6/precal_test_dataC/cal

to something appropriate on your own system you should be able to copy the "home" directory of these wrappers to your own computer and if you have the standalone pipeline installed on your computer, everything should work. I have done this between OSU and CTIO. (Depending on how you have things set up you may have to change an additional filepath or two, and I can help you if any problems occur.)