Project

General

Profile

Integrating DAGS with SAM - a simple example

This requires use of two KITS packages, jobsub_tools and ifdhc.

Ifdh has file copying, SAM manipulation, and logging utilities. All of these are demonstrated in these scripts.

%source /grid/fermiapp/products/(experiment)/etc/setups.sh
%setup jobsub_tools
%setup ifdhc

The DAG we will generate:


                         (start SAM project)

                        /        |         \
                       /         |          \
                  (consume   (consume    (consume
                     in         in          in
                    loop )     loop )      loop )
                      \          |           /
                       \         |          /
                        \        |         /

                      (end project and cleanup)

Here is the input file to dagNabbit.py which generates the DAG:

<macros>
$DATE_TAG=`date +%Y%m%d_%H%M%S_%N`
$PWD=`pwd`
$GRID_GROUP=`echo $GROUP`
$GRID_USER=`/usr/bin/whoami`
</macros>
<serial>
jobsub $PWD/pre.sh mwm_test_2 $DATE_TAG $GRID_GROUP $GRID_USER
</serial>
<parallel>
jobsub -g $PWD/middle.sh $DATE_TAG
jobsub -g $PWD/middle.sh $DATE_TAG
jobsub -g $PWD/middle.sh $DATE_TAG
</parallel>
<serial>
jobsub $PWD/post.sh $DATE_TAG
</serial>

pre.sh starts the SAM project:

#!/bin/sh -x
export DEFN=$1
export FILEBASE=$2
export GROUP=$3
export GRID_USER=$4
source /grid/fermiapp/products/$GROUP/etc/setups.sh
setup ifdhc
SOURCEFILE=$CONDOR_EXEC/dagtest/source_me.$FILEBASE.sh
EXP="export IFDH_BASE_URI=\"http://samweb-$GROUP.fnal.gov:20004/sam/$GROUP/api\"" 
eval $EXP
ifdh describeDefinition $DEFN
PROJNAME="${GRID_USER}_${FILEBASE}" 
echo "$EXP" > $SOURCEFILE
echo "export PROJNAME=$PROJNAME" >> $SOURCEFILE
echo "export GROUP=$GROUP" >> $SOURCEFILE
echo "export DEFN=$DEFN" >> $SOURCEFILE
echo "export GRID_USER=$GRID_USER" >> $SOURCEFILE
CPURL=`ifdh startProject $PROJNAME  $GROUP $DEFN  $GRID_USER $GROUP `
ifdh log "created project $PROJNAME consumer id $CONSUMER_ID host $HOSTNAME" 
echo "export CPURL=$CPURL" >> $SOURCEFILE
exit 0

middle.sh pulls files from the SAM project to the local grid node, and marks them consumed. This example only uses 3 consumers but any number >=1 will work.

#!/bin/sh 

function check
{
var=$@
echo "+ $var" 
eval $var
code=$?
if [ $code != 0 ]; then
        echo "$0: error: '$var' returned $code. exiting " 
        exit $code
fi
}

FILEBASE=$1
SOURCEME="$CONDOR_EXEC/dagtest/source_me.$FILEBASE.sh" 
check "source $SOURCEME" 
check "source  /grid/fermiapp/products/$GROUP/etc/setups.sh" 
setup ifdhc
check "CPURL=`ifdh findProject  $PROJNAME $GROUP `" 
check "CONSUMER_ID=`ifdh establishProcess $CPURL demo 1 $HOSTNAME $GRID_USER`" 
ifdh log "established consumer process $PROJNAME consumer id $CONSUMER_ID host $HOSTNAME" 
FLAG=true
check "FURI=`ifdh getNextFile $CPURL $CONSUMER_ID`" 
while [ "$FURI"  != "" ]
do
        check "FNAME=`ifdh fetchInput $FURI `" 
        check "ifdh updateFileStatus $CPURL  $CONSUMER_ID $FNAME transferred" 
        sleep 1
        check "ifdh updateFileStatus $CPURL  $CONSUMER_ID $FNAME consumed" 
        rm -f $FNAME
        FURI=`ifdh getNextFile $CPURL $CONSUMER_ID`
done
ifdh log "idfh out of files $PROJNAME consumer id $CONSUMER_ID host $HOSTNAME" 
#ifdh log does not return 0 ever
exit 0                                    

post.sh ends the project

#!/bin/sh -ex
FILEBASE=$1
SOURCEFILE=$CONDOR_EXEC/dagtest/source_me.$FILEBASE.sh
source $SOURCEFILE
source /grid/fermiapp/products/$GROUP/etc/setups.sh
setup ifdhc
source $SOURCEFILE
ifdh log "idfh ending project $PROJNAME host $HOSTNAME" 
ifdh endProject $CPURL
ifdh cleanup

Lets run the job:

[dbox@minervagpvm02 dagtest]$ dagNabbit.py -i input.dag.sam.2 -s
generated DAG saved as  /minerva/app/users/condor-tmp/dbox/submit.20120525_170143.dag
executing  ssh gpsn01.fnal.gov ". /opt/condor/condor.sh ; condor_submit_dag /minerva/app/users/condor-tmp/dbox/submit.20120525_170143.dag " 
to monitor your dags progress try this command:
  /grid/fermiapp/common/graphviz/zgrviewer/zgrview /minerva/app/users/condor-tmp/dbox/submit.20120525_170143.dag.dot &
[dbox@minervagpvm02 dagtest]$ 

Monitor the progress of your sam files at http://d0dbweb.fnal.gov:8080/station_monitor/minerva/stations/minerva/projects