Project

General

Profile

Submitting jobs

Project.py

Project.py is a python script that should make grid submission easy. You write an `xml` file and hand that some fcl files and input/ouput directories and then running the script begins running the jobs automatically. It also comes with some hand checking features to ensure the files aren't corrupt.

Example XML file

<!-- project details -->
<!DOCTYPE project [
<!ENTITY release "v06_26_01_08">
<!ENTITY name "v06_26_01_08_exampleScript">
<!ENTITY user "alister1">
<!ENTITY pers "/pnfs/uboone/persistent/users/alister1/exampleScript">
<!ENTITY app "/uboone/app/users/alister1/diffusion/prod/exampleScript">
]>

<project name="&name;">

    <!-- group -->
    <group>uboone</group>

    <!-- project size -->
    <numevents>10000</numevents>

    <!-- batchOS -->
    <OS>SL6</OS>

    <resource>DEDICATED,OPPORTUNISTIC,OFFSITE</resource>

    <!-- Here, we use a local uboonecode build. To do this, tar up your uboonecode version:
         -- Go to your top dir (where you have srcs/localproducts/build dirs)
         -- make_tar_uboone.sh some_tarred_filename.tar
         -- ifdh cp this file to your preferred dCache area -->
    <!-- larsoft info -->
    <larsoft>
        <tag>&release;</tag>    
        <qual>e10:prof</qual>
        <local>&pers;/ubcode_v06_26_01_08.tar</local>
    </larsoft>

    <!-- gen -->
    <stage name = "gen">
        <workdir>&pers;/gen_workdir</workdir>
        <outdir>&pers;/gen</outdir>
        <logdir>&pers;/gen</logdir>
        <fcl>&app;/prodcosmics_corsika_cmc_uboone.fcl</fcl>
        <bookdir>&app;/gen_bookdir</bookdir>
        <numjobs>600</numjobs>
    </stage>

    <!-- g4 -->
    <stage name = "g4">
        <workdir>&pers;/g4_workdir</workdir>
        <outdir>&pers;/g4</outdir>
        <logdir>&pers;/g4</logdir>
        <fcl>&app;/standard_g4_uboone.fcl</fcl>
        <bookdir>&app;/g4_bookdir</bookdir>
        <inputlist>&app;/inputg4.list</inputlist>
        <numjobs>600</numjobs>
        <memory>4000</memory>
    </stage>

    <!-- detsim -->
    <stage name = "detsim">
        <workdir>&pers;/detsim_workdir</workdir>
        <outdir>&pers;/detsim</outdir>
        <logdir>&pers;/detsim</logdir>
        <fcl>&app;/standard_detsim_uboone.fcl</fcl>
        <bookdir>&app;/detsim_bookdir</bookdir>
        <inputlist>&app;/inputdetsim.list</inputlist>
        <numjobs>600</numjobs>
        <memory>4000</memory>
    </stage>

    <!-- reco1 -->
    <stage name = "reco1">
        <workdir>&pers;/reco1_workdir</workdir>
        <outdir>&pers;/reco1</outdir>
        <logdir>&pers;/reco1</logdir>
        <fcl>&app;/reco_uboone_mcc8_driver_stage1.fcl</fcl>
        <bookdir>&app;/reco1_bookdir</bookdir>
        <inputlist>&app;/inputreco1.list</inputlist>
        <numjobs>600</numjobs>
    </stage>

    <!-- reco2 -->
    <stage name = "reco2">
        <workdir>&pers;/reco2_workdir</workdir>
        <outdir>&pers;/reco2</outdir>
        <logdir>&pers;/reco2</logdir>
        <fcl>&app;/reco_uboone_mcc8_driver_stage2.fcl</fcl>
        <bookdir>&app;/reco2_bookdir</bookdir>
        <inputlist>&app;/inputreco2.list</inputlist>
        <numjobs>600</numjobs>
    </stage>
</project>

The number of jobs is the total number of times the script will call the .fcl file and communicate with the grid. This translates to the number of output directories and output files you will have. The number of events is the total number of events you will generate. This means that you can calculate the number of events in a given file by simply taking `numevents/numjobs`.

Note that here we're using the "inputlist" tag, meaning that when your jobs have finished running you should ls -d /path/to/output/*.root > input{stagename}.list. If you don't use this you don't have to, but make sure to check the files (--check flag, see below) before you start the next stage.

How to run

project.py --xml xml_name.xml --stage stage_name --submit

and when that's finished, run
project.py --xml xml_name.xml --stage stage_name --check 

note that for any process which outputs an anatree file, `--check` should be replaced with `--checkana`. This will tell you if any processes returned errors. If they did then you can automatically regenerate them:
project.py --xml xml_name.xml --stage stage_name --makeup

If for some reason you need to submit the whole job again, you should either configure the .xml file to make a new repository, or you can get project.py to clean the repository for you by doing
project.py --xml xml_name.xml --stage stage_name --clean

The `--help` command for project.py is also pretty useful should you run into further problems.

To monitor your submitted jobs you can use the command `jobsub_q --user=user_name`.
Or to remove your jobs you can do `jobsub_rm --user=user_name --jobid=job_id`.