Feature #3150: Speeding up the FE matchmaking - pre-clustering
Reducing the memory footprint of the FE with clustering
The FE is currently loading all of the jobs in memory, just to ignore the details of most of them due to clustering!
We thus use a lot of memory; the CMS FE uses 1.5G per process... and we then fork this 5 times (not sure how much of that is shared).
Since the clustering criteria is known at startup time, we could cluster at condor_q (and possibly condor_status) time, discarding the details about the duplicates and just keeping the count.