Experiment-Defined Virtual Queues
We've been trying, with some difficulty, to coordinate our work around an external service which can only handle so much simultaneous load. --maxConcurrent helps, but it's been difficult to coordinate since there are multiple jobs, done by multiple people, which are all trying to use this same external resources. What we would really like is the ability to tell a job on submission that it belongs in some virtual queue, and we only allow X (ideally an easy to adjust number) of jobs in that queue to run simultaneously. This would allow us to limit the demand on this external resource while also allowing users to submit at any time without needing to coordinate with one another how many jobs they would run at the same time.