Job start staggering to reduce instaneous resource demand
I was wondering if there is currently the capability to stagger the start of certain types of jobs. Currently, for instance, I can submit a large number of calibration jobs which each access the database. I can stagger my submissions, but that doesn't guarantee they will start at different times due to the particular queue climate at the time of submission. What sometimes ends up happening is several hundred of these jobs hit the slots at the same time and end up waiting for their turn to access the DB. This causes huge efficiency losses if enough of these jobs start at the same time.
A solution is of course limit the number of jobs allowed to run at the same time. For my jobs in particular the DB access time is short in comparison to the processing time and happens all at once at run time. By limiting the number of jobs I can run at a time for fear of locking down a shared resource I'm increasing my full job set processing time by large amounts.
Could something like this be made available? It would greatly benefit job classes which use shared resources.