Quotas and subquotas on Fermigrid¶
Most FIFE experiments are assigned a quota of slots on Fermigrid. This quota is not the maximum amount of jobs that each experiments can run, but rather the number of slots that an experiment should get if every experiment is running at or above their quota.
Users can always see current allocations and subgroup breakout by looking at the "Modify Quota on Fermigrid" form here. This form, as the name implies, is used to request a change in the number of total slots for an experiment, and also their breakout amongst the subgroups.
As mentioned above, quotas are not a hard upper limit on the number of slots an experiment is allowed to run on Fermigrid. Rather, it is a minimum number of slots that the batch system will attempt to give an experiment if they have enough idle jobs waiting to run. This is best explained with an example.
If Nova has 2400 slots as its quota, has 3000 jobs waiting to run, and no jobs currently running, the batch system will try to hand out 2400 slots to Nova as they free up. There is no preemption on Fermigrid, so slots will not be shifted from one experiment to another immediately -- only as jobs finish running. Once the 2400 slots have Nova jobs running in them, the batch system will assign more slots to Nova only if other experiments are not using their entire quota. Similarly, if Nova has fewer than 2400 jobs running, and fewer than 2400 idle jobs, then as Nova jobs finish, the empty slots will be given to other experiments that have jobs waiting to run.
In the case where there are multiple experiments who would like to use more than their configured quotas, and one or more experiments that are not using their full quota, the excess from the latter set of quotas will be handed out to the former according to percentage of quota use: The less the percentage of a group's quota usage, the higher priority they have in receiving any excess slots.
Within a quota itself, jobs are handed out to users in order of priority. The more jobs a user runs, the higher this priority number becomes and their idle jobs will be considered later for matching to resources (allowing the job to run) by the batch system as compared to users with a lower priority number.Quota information can also be seen in these Fifemon pages:
- Experiment batch details: https://fifemon.fnal.gov/monitor/d/000000053/experiment-batch-details
- Fifebatch quotas (quotas of all experiments, along with high-level information about usage): https://landscape.fnal.gov/monitor/d/000000182/fifebatch-quotas
In addition to the quotas, experiments can subdivide their quota into subgroups (also known as subquotas). These are allocations of slots within a quota whose idle jobs are evaluated separately from the higher-level quota with respect to the decision of which jobs to run. In many cases, this can mean that jobs submitted to a subgroup will run before jobs submitted to the higher-level quota.
The details of subquotas is probably simpler to explain with an example. As of this writing, here is Nova's subdivision of their quota:
ana_high_prio = 0.15
ana_prio = 0.1
keepup_prio = 0.05
prod_high_prio = 0.5
prod_prio = 0.2
test = 0.01
The decimals next to the names are the the maximum percentage of the total quota (2400 slots at the moment) that each subgroup is allowed to take if the rest of the quota is filled. So, to take "prod_high_prio" as an example, that means that if Nova has 2400 slots filled with running jobs, and a Nova collaborator submits 1200 jobs to the prod_high_prio subgroup, the batch system will try to hand out 1200 slots to these new jobs first as old jobs finish and release those slots. Again, there's no preemption on Fermigrid, and so currently-running jobs won't ever be kicked out. If the total Nova quota is not filled, then just like the behavior of the higher level quotas, more than 1200 of these prod_high_prio jobs
could in principle run (the only subgroup this doesn't apply to is test -- only 1% of the slots will ever be given to test). Similarly, if less than 1200 prod_high_prio jobs are submitted, other subgroups could expand to take prod_high_prio's slots until more jobs were submitted to the prod_high_prio subgroup.
If multiple subgroups are submitted to, then the batch system tries to give slots to jobs whose subgroups are most under quota proportionally.
If users don't submit to any of these subgroups, then their jobs are simply considered under the main quota (2400 for Nova here), and should receive no advantage over other jobs (aside from priority, where the policy is simply to try to give resources first to those users who have used fewer resources as opposed to those who have used many). Jobs submitted to subgroups only compete against each other, and thus generally are considered first to run by the batch system.
To submit to a subgroup (let's use prod_high_prio again) simply add --subgroup prod_high_prio in the jobsub_submit command before the executable. So a sample jobsub_submit command might look like:
jobsub_submit -G nova --resource-provides=usage_model=DEDICATED --expected-lifetime='short' --subgroup prod_high_prio file:///path/to/my/executable