DES DM Stack Installation¶
This is a collection of notes from the initial installation of the DES DM software stack (version 6.3.2) at OSU.
Instructions from Brian and Marko with some feeback from OSU¶
SV team: Here are some notes from Marko Slyz on how to get the various 6.3.2 stack DESM codes and prereqs to compile on a stand alone Linux box running 'version 6' of redhat or scientific Linux. There is an older 'version 5' of Linux which we have also succeeded in getting to compile. SL5 and SL6 versions of Linux will be the 'supported' platforms, at least initially, and we will try and help you with compilation issues. ====================================================================== Contents: Building the DESDM 6.3.2 Stack Under EL6 Building Other Prerequisites (optional) Building the DESDM Home Code Prerequisites: You want to pick a top level directory, let's call this DES Once you have everything installed this directory will contain three subdirectories: DES_HOME for the DES codes such as mkbiascor stack for the DES DM 6.3.2 stack (e.g. swarp, scamp) desdm for the development code (wrappers) Building the DESDM 6.3.2 Stacks Under EL6 Note 1: These instructions are for the 6.3.2 stack. Note 2: Seem to need at least two cores to do the build, since otherwise the build code to handle the threading may get confused. 1. Setup eups: mkdir stack export DES_BASE='/n/des/desdm632 - but use your top level directory instead' export EUPS_PKGROOT=http://deslogin.cosmology.illinois.edu/desdmpkgs cd stack export DES_PREREQ=${DES_BASE}/stack curl -o desinstall.sh http://deslogin.cosmology.illinois.edu/desdmpkgs/desinstall.sh bash ./desinstall.sh eups source ${DES_PREREQ}/eups/1.1.1/bin/setups.sh 1.5 Some might find it convenient to combine some of these statements in a shell script since they will have to be executed frequently.
#!/usr/bin/env bash export SHELL='/bin/bash' export DES_BASE='/n/des/desdm632 - but use your top level directory instead' export EUPS_PKGROOT=http://deslogin.cosmology.illinois.edu/desdmpkgs export DES_PREREQ=${DES_BASE}/stack source ${DES_PREREQ}/eups/1.1.1/bin/setups.sh
2. Install extra packages if needed: yum install gcc-c++ zlib zlib-devel gcc-gfortranlibXft-devel libXpm libXpm-devel \ libXext-devel readline readline-devel libXft-devel gcc make Depending on your local setup this command might require root privileges 2.5 Cache credentials for svn: mv ~/.subversion ~/.subversion__original # Now there should be no .subversion directory present. # Do a random checkout just to cache the credentials: export DES_SVN=https://desweb.cosmology.illinois.edu/svn/desdm/devel svn co $DES_SVN/terapix/trunk terapix-trunk/psfex/eye/skymaker/psfex/src/fits # Accept the self-signed certificate permanently. # The fingerprint on mine is # 54:F1:74:56:BD:D3:AA:65:D8:BE:E8:BC:35:4B:BD:06:B6:D9:E7:A1 # Enter password, and say "yes" when it asks about storing password # unencrypted on disk. Another alternative is a keyring: # http://svnbook.red-bean.com/en/1.6/svn.serverconfig.netmodel.html#svn.serverconfig.netmodel.credcache 3. Start the build going until it produces an error: time eups distrib --noclean --install desdmstack 6.3.2 The --noclean switch here will leave the source code in place. 4. Several packages have known issues. These will cause the build script to stop (crash) nrecipes is the first of these stops that you will encounter Copy in a build script for nrecipes that has the -std=c89 switch. This switch should keep the system's fmin() function from being declared, which was conflicting with the nrecipes fmin() funtion. cp nrecipes-2.0.build stack/EupsBuildDir/Linux64/nrecipes-2.0/ # see attachment for this fixed nrecipes-2.0.build Now build nrecipes manually: cd stack/EupsBuildDir/Linux64/nrecipes-2.0/ bash ./nrecipes-2.0.build And "declare" it so that eups knows about it: time eups distrib --noclean --nobuild --install nrecipes 2.0 5. Try to build the rest of the stack until you run into the next problem: time eups distrib --noclean --install desdmstack 6.3.2 Here is a generic description of the procedures: During the build of some of the terrapix modules you might run into build and compile errors Most often this is caused by differences in system configuration. To fix this error go to the source directory of the product in question. Typically this can be found in stack/EupsBuildDir/Linux64/product-version/product-version/product For example, if you encounter an error building the psfex module you have to cd to stack/EupsBuildDir/Linux64/psfex-3.10.1/psfex-3.10.1/psfex/ cd to that directory and run autoreconf --force --install If your system doesn't have the autoreconf command you can get it from here: http://stackoverflow.com/questions/3096989/libtool-version-mismatch-error then check if your fix worked and try to build this module. Go to the module directory (stack/EupsBuildDir/Linux64/product-version) and run (for the psfex example) time eups distrib --noclean --nobuild --install psfex 3.10.1 Then go back to your top level directory and try again to build the complete DM stack time eups distrib --noclean --install desdmstack 6.3.2 You might have to follow this procedure for these products swarp scamp psfex sextractor 5.1 Fix for psfex (this is an application of the generic procedure described above. You do this only once per product) cd stack/EupsBuildDir/Linux64/psfex-3.10.1/psfex-3.10.1/psfex/ autoreconf --force --install cd ../.. time eups distrib --noclean --nobuild --install psfex 3.10.1 Or can probably just build the desdm stack. time eups distrib --noclean --install desdmstack 6.3.2 (this worked for me and I could skip the previous step) 5.2 Do the same thing for the sextractor: time eups distrib --noclean --install desdmstack 6.3.2 cd stack/EupsBuildDir/Linux64/sextractor-2.15.2/sextractor-2.15.2/sextractor/ autoreconf --force --install cd ../.. bash ./sextractor-2.15.2.build time eups distrib --noclean --install sextractor 2.15.2 Or can probably just build the desdm stack. time eups distrib --noclean --install desdmstack 6.3.2 (this worked for me and I could skip the previous step) 7. Finish the build: time eups distrib --noclean --install desdmstack 6.3.2 ---------------------------------------------------------------------- Building Other Prerequisites. These steps are optional. At OSU we used our existing Python 2.7.3 installation that already included numpy etc. 1. Install Python-2.7.x: # Do this in the same directory that has stack, and that will have DESDM home. # The page for the current (8/7/12) Python is http://www.python.org/getit/releases/2.7.3/ # Going mostly from memory for these commands, so they are somewhat approximate. wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tgz ./configure --prefix=`pwd` make make install # Make sure this is the Python that will be used: source setup_runB.sh . # set PATH to point to it python --version # May have to do something like chmod -R o+r Python-2.7.3 find Python-2.7.3 -executable -exec chmod o+x {} \; # to share with other users, but don't know the # reason for the restrictive permissions. 2. Building numpy: wget http://downloads.sourceforge.net/project/numpy/NumPy/1.6.2/numpy-1.6.2.zip? r=http%3A%2F%2Fsourceforge.net%2Fprojects%2Fnumpy%2Ffiles%2FNumPy%2F1.6.2%2F&ts=1343793172&use_mirror=voxel export PYTHONPATH= # untar cd /cloud/login/mslyz/codes_try5b/numpy-1.6.2 python setup.py build --fcompiler=gnu95 python setup.py install or just type pip install numpy 3. Building pyfits: # Following instructions at http://www.stsci.edu/institute/software_hardware/pyfits/Download : wget http://pypi.python.org/packages/source/p/pyfits/pyfits-3.0.8.tar.gz export PYTHONPATH=${DES_BASE}/Python-2.7.3/lib/python2.7/site-packages tar xf pyfits-3.0.8.tar.gz cd ~/codes_try5b/pyfits-3.0.8/ python setup.py install or just type pip install pyfits # When done there should be numpy and pyfits directories in # /cloud/login/mslyz/codes_try5b/Python-2.7.3/lib/python2.7 ---------------------------------------------------------------------- Building the DESDM Home Code These programs are written specifically for DES. They link against the libraries in the stack, so the stack has to be installed first. 1. Check out the code: cd $DES_BASE # go to your top level directory mkdir -p desdm/devel/ImageProc cd desdm/devel/ImageProc svn checkout https://desweb.cosmology.illinois.edu/svn/desdm/devel/ImageProc/trunk 2. Fix Makefile in desdm/devel/ImageProc/trunk/src/Makefile to just compile the needed programs: imcorrect DECam_crosstalk fwhm runSExtractor fitscombine mkbiascor mkflatcor mkillumcor mksupersky 3. Run source setup_build.sh . 4. Do the actual compilation: cd desdm/devel/ImageProc/trunk/ make make install 5. Check out the Wrappers and Integration code. This is all scripts, and doesn't need building. cd desdm/devel svn co https://desweb.cosmology.illinois.edu/svn/desdm/devel/Wrappers svn co https://desweb.cosmology.illinois.edu/svn/desdm/devel/IntegrationUtils ================
The replacement files and scripts you need for the stack and des code installation are available from this page. Please note that there might be newer versions of these files available directly from DES DM.