Project

General

Profile

Idea #3160

Feature #3150: Speeding up the FE matchmaking - pre-clustering

Reducing the memory footprint of the FE with clustering

Added by Igor Sfiligoi over 6 years ago. Updated over 4 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
Frontend
Target version:
Start date:
12/04/2012
Due date:
% Done:

0%

Estimated time:
Stakeholders:

CMS

Duration:

Description

The FE is currently loading all of the jobs in memory, just to ignore the details of most of them due to clustering!

We thus use a lot of memory; the CMS FE uses 1.5G per process... and we then fork this 5 times (not sure how much of that is shared).

Since the clustering criteria is known at startup time, we could cluster at condor_q (and possibly condor_status) time, discarding the details about the duplicates and just keeping the count.

History

#1 Updated by Igor Sfiligoi over 6 years ago

This change would likely need the refactoring of several functions in several files.

#2 Updated by Igor Sfiligoi over 6 years ago

Further update:
System memory usage goes from 2G to 6.5G when all 5 processes are running.

So, for all practical purposes, CMS needs ~5G to run the FE.

#3 Updated by Burt Holzman over 6 years ago

  • Target version set to v3_1

#4 Updated by Burt Holzman over 6 years ago

  • Target version changed from v3_1 to v3_x

#5 Updated by Parag Mhashilkar over 4 years ago

  • Stakeholders updated (diff)

#6 Updated by Parag Mhashilkar over 4 years ago

  • Assignee deleted (Igor Sfiligoi)

Igor Sfiligoi's tickets. These were some ideas brewing up in his mind but they never materialized or got priority.



Also available in: Atom PDF