Project

General

Profile

Feature #5653

Add support for multiple collector partitions

Added by Igor Sfiligoi over 5 years ago. Updated over 4 years ago.

Status:
New
Priority:
Low
Assignee:
-
Category:
Frontend
Target version:
Start date:
03/13/2014
Due date:
% Done:

0%

Estimated time:
Stakeholders:

CMS

Duration:

Description

Right now, the FE assumes all glideins will go into the same Condor pool, served by a single top collector.

For scalability reasons, CMS wants to (randomly) partition the glideins into multiple pools (i.e. top Collectors),
so the FE and glideins need to gain this capability.


Related issues

Related to GlideinWMS - Feature #5652: Add support for flocked collectorsNew03/13/2014

History

#1 Updated by Igor Sfiligoi over 5 years ago

#2 Updated by Igor Sfiligoi over 5 years ago

  • Stakeholders updated (diff)

#3 Updated by Burt Holzman over 5 years ago

This feature implies we've given up having HTCondor improve the system scalability.

#4 Updated by Igor Sfiligoi over 5 years ago

Improvements in HTCondor-land would obviously be welcome, but this allows us to add another dimension to the scalability in addition to whatever HTCondor provides.

#5 Updated by Igor Sfiligoi over 5 years ago

My proposal is to add a partition attribute to the collector element;
just like we have the group attribute now.

So CMS would end up with something like this:

   <collectors>
      <collector DN="DN1" group="c1" partition="p1" node="glidein-collector.t2.ucsd.edu:9618" secondary="False"/>
      <collector DN="DN1" group="c1" partition="p1" node="glidein-collector.t2.ucsd.edu:9620-10119" secondary="True"/>
      <collector DN="DN2" group="c2" partition="p1" node="glidein-collector-2.t2.ucsd.edu:9618" secondary="False"/>
      <collector DN="DN2" group="c2" partition="p1" node="glidein-collector-2.t2.ucsd.edu:9620-10119" secondary="True"/>
      <collector DN="DN1" group="c1" partition="p2" node="glidein-collector.t2.ucsd.edu:19618" secondary="False"/>
      <collector DN="DN1" group="c1" partition="p2" node="glidein-collector.t2.ucsd.edu:19620-20119" secondary="True"/>
      <collector DN="DN2" group="c2" partition="p2" node="glidein-collector-2.t2.ucsd.edu:19618" secondary="False"/>
      <collector DN="DN2" group="c2" partition="p2" node="glidein-collector-2.t2.ucsd.edu:19620-20119" secondary="True"/>
   </collectors>

Let me know if you don't like it.

#6 Updated by Parag Mhashilkar over 5 years ago

Please do not start implementing. We need to think over this and discuss it.

#7 Updated by Igor Sfiligoi over 5 years ago

  • Priority changed from Normal to Low
  • Target version changed from v3_2_5 to v3_x

There is a proposal in HTCondor to use secondary collectors for all classad needs. That would remove the need for multiple "top collectors".
CMS likes the idea and wants to use it.

I still think it would be a good idea to support multiple partitions (e.g. to get multiple Negotiators), but it is definitely not a priority anymore (for CMS).

#8 Updated by Parag Mhashilkar over 4 years ago

  • Assignee deleted (Igor Sfiligoi)

Igor Sfiligoi's tickets. These were some ideas brewing up in his mind but they never materialized or got priority.



Also available in: Atom PDF