Project

General

Profile

Wiki

The ifdh_art package provides ART service access to the libraries from the ifdhc package.

The usage of the packages is largely the same, except that you use ART service handles to access the
various types.

Easy Usage

If you are running Art framework jobs at Fermilab with Jobsub, and you have suitable sam metadata configuration in your .fcl files (see the next section for details), you can just submit jobs with jobsub, with an invocation like:

jobsub \
   -g  \
  -r S12-11-16 \
  -N 6  \
  --dataset_definition=my_dataset_name \
  $IFDH_ART_DIR/bin/art_sam_wrap.sh  \
      -X nova  \
      -c /path/to/my/config.fcl \
      --rename uniq \
      --dest /grid/data/$USER 

This 1) submits a DAG of jobs which 2) are grid jobs, and 3) use release S12-11-16, 4) has 6 nodes running to analyse the files in the 5) my_dataset_name dataset using 6) the art_sam_wrap.sh wrapper to run the art 7) nova executable with a 8) specified fcl file and asks 9) that the output file be renamed to be unique, and 10) copied back to the users nova data area. You can see How it should look, but if your DAG fizzles, see Debugging SAM job sumbissions.

If instead you wanted to run something like a reconstruction set, where you want exactly one output file per input file, at the moment you need to launch one job per file, and that looks like this:

jobsub \
  -g  \
  -r S12-11-16 \
  -N 100  \
  --dataset_definition=my_dataset_name  \
  $IFDH_ART_DIR/bin/art_sam_wrap.sh  \
      -X nova  \
      -c /nova/app/users/mengel/submit_me.fcl \
      --rename s/raw/reco/ \
      --dest /grid/data/$USER \
      --limit 1 \

This 1) submits a DAG of jobs which 2) are grid jobs, and 3) use release S12-11-16, 4) has 100 jobs running to analyse the files in the 5) my_dataset_name dataset using 6) the art_sam_wrap.sh wrapper to run the art 7) nova executable with a 8) specified fcl file and asks 9) that the output file be renamed to be the same as the input file with "raw" replaced by "reco", and 10) copied back to the users nova data area, and 11) only deliver 1 file per job.

This assumes that the "my_dataset_name" has 100 files in it -- you can look this up in the web dataset editor.

You can see How it should look; if your DAG fizzles, see Debugging SAM job sumbissions.

fhcl files

If you are using these services, you need to mention them in your .fcl files; although if you use the --sam-* command line options, many of these entries will get added to your configuration automatically; and the art_sam_wrapper.sh script will similarly add some of these options to your config before running
your program.

An example with configuration for all of the services is:


outputs.out1.dataTier: "raw" 
process_name: "whatever" 

services: {

  FileCatalogMetadata: { 
     applicationFamily: "demo" 
     applicationVersion: "1" 
     fileType: "importedDetector" 
  } 

  user: {

    # if using ART with SAM, you need these entries, OR the --sam-* command-line options

    IFDH: { 
     IFDH_BASE_URI: "http://samweb.fnal.gov:8480/sam/minerva/api" 
    }

    CatalogInterface: { service_provider: "IFCatalogInterface" }
    FileTransfer: { service_provider: "IFFileTransfer" }

    # if you use either of IFBeam or nucondb services, include the respective
    # entries below.
    IFBeam: {}
    nucondb: {}
  }

}

or as one-liners:

# if using ART with SAM, you need these entries, OR the --sam-* command-line options
user.services.IFDH: {IFDH_BASE_URI: "http://samweb.fnal.gov:8480/sam/minerva/api"}
user.services.CatalogInterface: { service_provider: "IFCatalogInterface" }
user.services.FileTransfer: { service_provider: "IFFileTransfer" }
outputs.out1.dataTier: "raw" 
process_name: "whatever" 
services.FileCatalogMetadata: { applicationFamily: "demo" 
                                applicationVersion: "1" 
                                fileType: "importedDetector" 
                              } 
# if you use either of IFBeam or nucondb services, include the respective
# entry below.
user.services.IFBeam: {}
user.services.nucondb: {}

Note that the IFCatalogInterface and the IFFileTransfer requirethe IFDH service as well, so you must include IFDH if you use the others.

You can omit the IFDH_BASE_URI: "http:..." part if IFDH_BASE_URI is set in the environment.

Using IFDH service directly

If you need to access multiple datasets, or look up file locations, or other things than simply get and read the next file in a dataset in your art framework software, you need the IFDH service, which is the Art service for the ifdh.h code from ifdhc. You use:

#include "IFDH_service.h" 
art::ServiceHandle<IFDH> ifdhp;

instead of

#include "ifdh" 
std::unique_ptr<ifdh> ifdhp( new ifdh());

but otherwise you use the the ifdh layer as documented in it's wiki

Note that you probably won't use this service directly, unless you're doing something complicated. Usually (if your job is a single consumer of a SAM project) you will use this service indirectly via IFFileCatalog and IFFileTransfer, below.

However if you need to consume files from multiple SAM projects, start new SAM projects within a job, or you want to have multiple file transfers going simultaneously, etc. you would use this service directly. You might also use this service to transfer ancillary files that are not exactly usual input or output files.

IFBeam service

The IFBeam service is the Art service for the ifbeam.h beams database access code from ifdhc. You use:

#include "IFBeam_service.h" 
art::ServiceHandle<IFBeam> ifbeam_handle; 
std::unique_ptr<BeamFolder> bfp( ifbeam_handle->getBeamFolder(bundle, url, width));

instead of

#include <ifbeam.h>
unique_ptr<BeamFolder> bfp( new BeamFolder(bundle, url, width));

but otherwise you use the the ifbeams layer as documented in it's wiki

NuConDB service

The NuConDB service is the Art service for the nucondb.h conditions database code from ifdhc. You use:

#include "nucondb_service.h" 
art::ServiceHandle<nucondbService> nucondb_handle; 
std::unique_ptr<Folder> pf( nucondb_handle->getFolder(name, url, tag));

instead of

#include "nucondb.h" 
std::unique_ptr<Folder> pf(new Folder("minerva_atten_id", "http://dbweb0.fnal.gov:8088/mnvcon_int/app","old"));

but otherwise you use the the nucondb layer as documented in it's wiki

IFCatalogInterface and IFFileTransfer

These services are implementations of general interfaces in ART which areinstantiated for folks using SAM data handling. You don't need to specifically call them, only mention them in your .fcl files as mentioned above.