PBS Maxwalltime for long running pilots
Purdue has requested that the factory only send multicore pilots. Purdue wants these pilots to have a long lifetime (on the order of 5 days). However, when the site admins put a future downtime into PBS, PBS will dynamically shorten the maxwalltime allowed for a batch job as it gets closer to the downtime. Since the factory has hardcoded RSL, this means that once PBS starts to shorten the maxwalltime, pilots will be submitted and queued on the site, but will remain in idle state since they are requesting a maxwalltime greater than what PBS is willing to give them. The question is what if anything we can do to maximize the time that pilots can run.