Feature #5653

Add support for multiple collector partitions

Added by Igor Sfiligoi over 6 years ago. Updated about 2 months ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:




Right now, the FE assumes all glideins will go into the same Condor pool, served by a single top collector.

For scalability reasons, CMS wants to (randomly) partition the glideins into multiple pools (i.e. top Collectors),
so the FE and glideins need to gain this capability.

Related issues

Related to GlideinWMS - Feature #5652: Add support for flocked collectorsNew03/13/2014


#1 Updated by Igor Sfiligoi over 6 years ago

#2 Updated by Igor Sfiligoi over 6 years ago

  • Stakeholders updated (diff)

#3 Updated by Burt Holzman over 6 years ago

This feature implies we've given up having HTCondor improve the system scalability.

#4 Updated by Igor Sfiligoi over 6 years ago

Improvements in HTCondor-land would obviously be welcome, but this allows us to add another dimension to the scalability in addition to whatever HTCondor provides.

#5 Updated by Igor Sfiligoi over 6 years ago

My proposal is to add a partition attribute to the collector element;
just like we have the group attribute now.

So CMS would end up with something like this:

      <collector DN="DN1" group="c1" partition="p1" node="" secondary="False"/>
      <collector DN="DN1" group="c1" partition="p1" node="" secondary="True"/>
      <collector DN="DN2" group="c2" partition="p1" node="" secondary="False"/>
      <collector DN="DN2" group="c2" partition="p1" node="" secondary="True"/>
      <collector DN="DN1" group="c1" partition="p2" node="" secondary="False"/>
      <collector DN="DN1" group="c1" partition="p2" node="" secondary="True"/>
      <collector DN="DN2" group="c2" partition="p2" node="" secondary="False"/>
      <collector DN="DN2" group="c2" partition="p2" node="" secondary="True"/>

Let me know if you don't like it.

#6 Updated by Parag Mhashilkar over 6 years ago

Please do not start implementing. We need to think over this and discuss it.

#7 Updated by Igor Sfiligoi over 6 years ago

  • Priority changed from Normal to Low
  • Target version changed from v3_2_5 to v3_x

There is a proposal in HTCondor to use secondary collectors for all classad needs. That would remove the need for multiple "top collectors".
CMS likes the idea and wants to use it.

I still think it would be a good idea to support multiple partitions (e.g. to get multiple Negotiators), but it is definitely not a priority anymore (for CMS).

#8 Updated by Parag Mhashilkar over 5 years ago

  • Assignee deleted (Igor Sfiligoi)

Igor Sfiligoi's tickets. These were some ideas brewing up in his mind but they never materialized or got priority.

#9 Updated by Marco Mascheroni about 2 months ago

  • Status changed from New to Closed

Also available in: Atom PDF