Project

General

Profile

Bug #23947

Postscript-unquote options are ignored when running fife_wrap in v3_3

Added by Kenneth Herner 4 months ago. Updated about 2 months ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
Start date:
01/26/2020
Due date:
% Done:

0%

Estimated time:
Duration:

Description

Hi,

It appears that while postscript-unquote options do get parsed properly in v3_3, they don't actually end up running in the job. I've confirmed that in my output and it also shows up if you use the --dry_run option. Here is an example using /dune/app/home/dunepro/poms_MC_Fall2019/RITM0892689_ndk/mergeana.cfg, with v3_3 set up:

# setup commands:
# I would run:
for p in /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups /grid/fermiapp/products/common/etc/setups ; do if [ -r $p ]; then . $p; break; fi; done
export FILETIMESTAMP=$(date +%Y%m%dT%H%M%S)
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunetpc v06_85_00 -q e15:prof
setup dune_oslibs v1_0_0
[ -z "$PRODUCTS" -a -r /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh ] && source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
[ -z "$PRODUCTS" -a -r /grid/fermiapp/products/common/etc/setups.sh ] && source /grid/fermiapp/products/common/etc/setups.sh
[ -z "$IFDHC_DIR" ] && source `ups setup ifdhc`
# end setup commands
# note: later commands are paraphrased from python
# I would exec python again... with --nosetup
cpurl=`ifdh findProject "$SAM_PROJECT" "dune"`
consumerid=`ifdh establishProcess "$cpurl" "ana" "v06_85_00" "dunegpvm15.fnal.gov" "dunepro" "art" "process 6330 on dunegpvm15.fnal.gov" "20" "root"`
# pre-scripts-quoted
# I would run:
for fclfile in gen.fcl; do if [ ! -f ${fclfile} ]; then cp ${DUNETPC_DIR}/job/${fclfile} . ; fi ; echo 'services.FileCatalogMetadata.runType: "fardet-sp"' >> ${fclfile} ; done
lar --sam-web-uri=$cpurl --sam-process-id=None -c gen.fcl -n 1 -o generic_${CLUSTER}_${PROCESS}_TIMESTAMP_merged.root -T generic_${CLUSTER}_${PROCESS}_TIMESTAMP_anatree.root
rres=$?
        ifdh addOutputFile *anatree*.root
I would get base metadata for *anatree*.root
        # ifdh copyBackOutput  /path/to/dest 0
I would clean up output files
if [ \! -z "$cpurl" -a \! -z "$consumerid" ]
then
    if [ $rres = 0  ]
    then
        ifdh setStatus "$cpurl" "$consumerid" "completed" 
    else
        ifdh setStatus "$cpurl" "$consumerid" "bad" 
    fi
fi
exit $rres
trying to exit 0

But now look at the same command when run with v3_2_9:

# setup commands:
# I would run:
for p in /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups /grid/fermiapp/products/common/etc/setups ; do if [ -r $p ]; then . $p; break; fi; done
export FILETIMESTAMP=$(date +%Y%m%dT%H%M%S)
source /cvmfs/dune.opensciencegrid.org/products/dune/setup_dune.sh
setup dunetpc v06_85_00 -q e15:prof
setup dune_oslibs v1_0_0
[ -z "$PRODUCTS" -a -r /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh ] && source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
[ -z "$PRODUCTS" -a -r /grid/fermiapp/products/common/etc/setups.sh ] && source /grid/fermiapp/products/common/etc/setups.sh
[ -z "$IFDHC_DIR" ] && source `ups setup ifdhc`
# end setup commands
# note: later commands are paraphrased from python
# I would exec python again... with --nosetup
cpurl=`ifdh findProject "$SAM_PROJECT" "dune"`
consumerid=`ifdh establishProcess "$cpurl" "ana" "v06_85_00" "dunegpvm15.fnal.gov" "dunepro" "art" "process 5151 on dunegpvm15.fnal.gov" "20" "root"`
# pre-scripts-quoted
# I would run:
for fclfile in gen.fcl; do if [ ! -f ${fclfile} ]; then cp ${DUNETPC_DIR}/job/${fclfile} . ; fi ; echo 'services.FileCatalogMetadata.runType: "fardet-sp"' >> ${fclfile} ; done
lar --sam-web-uri=$cpurl --sam-process-id=None -c gen.fcl -n 1 -o generic_${CLUSTER}_${PROCESS}_TIMESTAMP_merged.root -T generic_${CLUSTER}_${PROCESS}_TIMESTAMP_anatree.root
rres=$?
        ifdh addOutputFile *anatree*.root
I would get base metadata for *anatree*.root
        # ifdh copyBackOutput  /path/to/dest 0
Here 0 alloutputs: []
I would clean up output files
# I would run:
cp ${JSB_TMP}/JOBSUB_LOG_FILE ./generic_gen_g4_${CLUSTER}_${PROCESS}.out
cp ${JSB_TMP}/JOBSUB_ERR_FILE ./generic_gen_g4_${CLUSTER}_${PROCESS}.err
mv $(ls *merged.root) $(echo $(ls *merged.root) | sed -e "s/TIMESTAMP/${FILETIMESTAMP}/")
mv $(ls *anatree.root) $(echo $(ls *anatree.root) | sed -e "s/TIMESTAMP/${FILETIMESTAMP}/")
extractor_prod.py --infile $(ls *merged.root) --no_crc --appname anatree --appversion v06_85_00 --campaign override_me --appfamily art | sed -r -e "/file_format/ s/artroot/rootntuple/" -e "/data_tier/ s/full-reconstructed/root-tuple/" -e "/file_size/ s#\:.*#\: $(ls -l *anatree*.root | awk '{print $5}'),#" -e "/art.file_format/d" -e "/file_name/ s/:.*merged.root\",/: \"$(ls *anatree*.root)\",/" > $(ls *anatree.root).json
if [ \! -z "$cpurl" -a \! -z "$consumerid" ]
then
    if [ $rres = 0  ]
    then
        ifdh setStatus "$cpurl" "$consumerid" "completed" 
    else
        ifdh setStatus "$cpurl" "$consumerid" "bad" 
    fi
fi
exit $rres
trying to exit 0

Here, we DO see the postscript commands (2 cp, 2 mv, and extractor_prod.py). Also it appears that at least in v3_2_9 things are still running after the file copyback (which won't work because the metadata extractor needs to run first). The old older was executable, postscript, copyback. This is with the ifdh_art option set if that makes any difference.

Ken

History

#1 Updated by Marc Mengel about 2 months ago

  • Status changed from New to Resolved

Fixed in d0f5c5ca.

#2 Updated by Marc Mengel about 2 months ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF