Project

General

Profile

Nearline Monitoring - Experts Only

Misc Links

You can check the status of the POMS Project for Electron Lifetime: https://pomsgpvm01.fnal.gov/poms/campaign_stage_info?campaign_stage_id=42. The campaign runs on uboonegpvm05 and was setup by SCD, and handed off to us. You will need to be setup with permissions if you want to edit/start/stop the campaign (contact POMS admins).

The code for the analysis portion of the nearline monitoring is on github: https://github.com/mibass/ubooneNearLine. The descriptions for each of the scripts is in the readme.md there. There is also an example ipynb showing how to analyse the sqlite db.

This runs every four hours in a cron job on uboonegpvm01 as uboonepro. The cron calls the runs.sh script and stores the generated logs in /uboone/app/home/uboonepro/NearLine/cronlog.{log, err}.

If the plots on the elog are failing to update, check the following:
  • Is the cron job running? (Check the timestamps on the log files.)
  • Check the log files. Are there any errors? Are there new files being found by the Tracker.py script?
  • Is the campaign finding new files? Go to the above link for the POMS campaign and check the output of recent runs.

If plots are not posted on the elog for over 8 hours, log on gpvm and cd /uboone/app/home/uboonepro/NearLine/, use the following command to check whether there is any stucked runs.sh running:
ps ax | grep runs.sh | grep -v $$ | grep bash | grep -v grep
If yes, Kill the stuck process(es).

TODO:
  • The sqlite db is not currently backed up. In principle it is possible to reprocess and regenerate the db from files on disk, but this could take a few days with the current setup. It would be nice to back up the database, possibly into the calib db?