Project

General

Profile

Data and MC Production

Jobs are submitted as user lariatpro from a dedicated virtual machine lariatsamgpvm01.fnal.gov, the Production VM.

Raw Data: Keeping up with new data

Campaign stage Launch Template Job Type comment
LArIAT Raw2Digits setup lariat LArIAT test Old style and complicated.
script: /lariat/app/home/lariatpro/KeepUp/ProductionKeepUp_LArIAT_20160820.sh

~lariatpro/KeepUp/ is the stuff for the old, reliable digit-making job.

The POMS website sets a crontab on the VM, specifying how often and when to run a certain script, specified on the site.
  • I think the script we want here is /lariat/app/home/lariatpro/KeepUp/ProductionKeepUp_LArIAT_20161103.sh right now.
  • It's kinda fancy, but really it's getting the lists of files from complete runs more than 24 hours old, then running the digit-making fcl file, running a little script to whip up JSON metadata files for each of them, and stashing the results.

What's nice about POMS is that it handles the cron jobs to launch them, then also tracks all the jobs, one per run, and categorizes the failures by error code, and provides access to the log files, etc.

When the job returns a file of digits or whatever, along with a tiny JSON file of the file's metadata, we want it to get moved to permanent storage and then declared to SAM.
So that's managed in the FTS directory. Probably we won't mess much with that. IT's already working well.

Raw Data: Reprocessing existing data

This is a work in progress. Want to be able to reprocess all old data whenever the raw2digit stuff gets a significant upgrade.

If you want to update a campaign, it's best practice to make a clone of the existing campaign. For example, let's say we want to use a newer version of lariatsoft (ie, v06_33_00 -> v06_34_00). We would then:
  1. Go to POMS https://pomsgpvm01.fnal.gov/poms/
  2. Go to Compose Campaign Stages and clone Raw2Digits_Reprocessing_v06_33_00
  3. Rename to “xxx_v06_34_00” and remove “CLONE OF:"
  4. Change version to v06_34_00.
  5. Go to "Param Overrides" and change process name to match the campaign name.

Getting the processed files stored permanently and registered in SAM: File Transfer System for MC and Processed Data

User lariatpro is running an instance of Fermilab's FTS (File Transfer System). It scans certain directories for new files and their JSON metadata files, moving the files to their permanent locations on tape-backed pnfs "DCache" storage and declaring the files to the SAM metadata file catalog.

Where is it?

The FTS instance runs on a dedicated gpvm (General Purpose Virtual Machine). Your kerberos account must be listed in the lariatpro .k5login file. And then you can do this:

ssh -XY lariatpro@lariatsamgpvm01.fnal.gov
cd /lariat/app/home/lariatpro/fts/lariatsamgpvm01.fnal.gov

What can it do?

Details about FTS: https://cdcvs.fnal.gov/redmine/projects/filetransferservice/wiki
FTS configuration file contents

What if there's trouble?

When it is running, the FTS instance hosts a nice little monitoring page: http://lariatsamgpvm01.fnal.gov:8787/fts/status

Changes to the configuration take effect only when FTS is re-started:

cd /lariat/app/home/lariatpro/fts/lariatsamgpvm01.fnal.gov
stop_fts ./
start_fts ./ lariatsamgpvm01.fnal.gov_fts_config.ini