Project

General

Profile

Cron jobs for creating JSON metadata files on the np04 cluster

For reference, here are my elog posts on this subject: [to see older elog posts, use the Advanced Search]

07-Dec-2017
I have installed a cron job on np04-srv-014 that will periodically check if new raw
data files have appeared in /nfs/sw/data, and if so, create companion JSON metadata
files in the same directory.
The future intention of these metadata files is to serve as indicators that the raw
data files are ready to be copied downstream. These metadata files should not affect
the running of the DAQ, and they won't actually initiate any sort of file copy until
additional steps are taken to install the data-copying tools (not sure when that might
happen). This script is simply being tested now.

At the moment, the contents of the metadata files are fairly basic; they just have a
few parameters to describe the associated raw data file. The naming and contents of the
metadata files may change in the future.

The script that is being run by the cron job is /nfs/home/np04daq/.cron/createMetadataFile.sh.
The crontab file that is used to register the cron job is /nfs/home/np04daq/.cron/np04-14.crontab.
The script creates log file entries in np04-srv-014:/log/metadataFileCreator/createMDFile.log.

There are tunable parameters at the top of the createMetadataFile.sh script that can be
changed to modify its behavior. Please save a backup copy if you change the script.

To un-register the cron job, you can use 'crontab -r' (run from the np04daq account on
np04-srv-014).
Sunday, 27-Jan-2018
Subject: crontab created on np04-srv-001 to create metadata files for raw data files

To do this, I copied the createMetadataFile.sh script to createMetadataFile01A.sh in
/nfs/home/np04daq/.cron and modified the new "01A" copy to look at the /data0 disk
(and create lock and log files with "01A" in their name).
I then created the /nfs/home/np04daq/.cron/np04_01.crontab file and used it to create
the cronjob ('crontab np04_01.crontab).

Sunday, 28-Jan-2018
I also created a directory for log files:  /log/metadataFileCreator