Crontabs run as uboonepro¶
There are a sequence of cron jobs run on the interactive nodes using the uboonepro account. These cron jobs can be located from here:
where XX is the number of the interactive node [01,02,03,04,05,06,07]. To get a listing of the current cron jobs running on a node, log into the node as uboonepro and run this command:
$> ssh -Y firstname.lastname@example.org [uboonepro@uboonegpvm07 cron]$ crontab -l
In general, the jobs are run from the /uboone/app/uboonepro/cron/ directory, and they keep output and error logs in files of the format cron_<task>.out and cron_<task>.err where task is something like "kill_held_jobs".
Checking the Database status¶
In order to make sure that databases are running properly, there is a cron job that tries a simple connection to the IFBeam, calibration production, and calibration development databases. If any of these checks fails, then the output of the check is emailed to experts. Here is the actual entry in the crontab for email@example.com
15 * * * * cd /uboone/app/home/uboonepro/cron;./cron_check_database.sh > cron_check_database.out 2> cron_check_database.err
So this check of the databases happens once an hour at 15 minutes past the hour.
Killing held and long jobs¶
In order that the uboonepro account does not have an excessive number of held jobs left in the jobsub server that would block the subsequent submission of additional jobs, once an hour at 1 minute past the hour, there is a crontab job run to kill all held jobs for the uboonepro account on the jobsub server. Additionally, there is another crontab job that kills long running jobs every hour at 30 minutes past the hour.
1 * * * * cd /uboone/app/home/uboonepro/cron;./cron_kill_held_jobs.sh > cron_kill_held_jobs.out 2> cron_kill_held_jobs.err 30 * * * * cd /uboone/app/home/uboonepro/cron;./cron_kill_long_jobs.sh > cron_kill_long_jobs.out 2> cron_kill_long_jobs.err