Project

General

Profile

Feature #2560

Support multiple collectors in HA mode

Added by Parag Mhashilkar almost 8 years ago. Updated about 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Parag Mhashilkar
Category:
-
Target version:
Start date:
08/14/2012
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Stakeholders:
Duration:

Description

The CMS collector node at CERN is misbehaving, and they would have liked to use two collector nodes.

As you probably know, this is currently not possible (due to the collector selection code in the glideins).
I think we should make this high priority.

My 2c,
Igor


Subtasks

Support #2882: Helping the VOs install a HA setupClosedJohn Weigand

History

#1 Updated by Burt Holzman almost 8 years ago

  • Target version changed from v2_5_7 to v2_7_x

#2 Updated by Parag Mhashilkar almost 8 years ago

I gave this a considerable thought. In the end all we need to do is set HEAD_NODE and CCB_ADDRESS a list of HA collectors

The changes can either be just head_node selection algorithm in glidein or it would also include support & distinction for HA collectors in the frontend.xml

If we want to keep it simple and focused on head_node selection by glidein here is my proposal:

  • All the non-primary HA collectors are added with secondary="True" * Change the head_node selection logic to select a list of all the head nodes. However only one is selected per hostname to cover for non-contiguous range of secondary collector ports

Only caveat in this is if a VO decides to come up with a messed up setup where main HA collector does not have secondary collectors but other HA collectors do. Any one in right mind wouldn't do this but still for this deployment model an easy fix is have the primary collector entry listed as a secondary as well.

Is this acceptable?

#3 Updated by Igor Sfiligoi almost 8 years ago

I think we want to be more explicit.
While you proposal seems reasonably good, it would not cover for the use can when you want to force HA over the altantic, but still have multiple Collector hosts on each coast.

I would add instead a "group" attribute, and the glidein then picks out a collector from each group.
I would also require the admin to explicitly all the groups, to prevent simple typos (and possibly to make it easy to exclude a group, if needed).

Here is an example of my proposal:

 
  <collectors groups="1,2">
      <collector DN="/DC=org/DC=doegrids/OU=Services/CN=glidein-collector.t2.ucsd.edu" node="glidein-collector.t2.ucsd.edu" secondary="False" group="1"/>
      <collector DN="/DC=org/DC=doegrids/OU=Services/CN=glidein-collector.t2.ucsd.edu" node="glidein-collector.t2.ucsd.edu:9620-9919" secondary="True" group="1"/>
      <collector DN="/DC=ch/DC=cern/OU=computers/CN=vocms105.cern.ch" node="vocms105.cern.ch" secondary="False" group="2"/>
      <collector DN="/DC=ch/DC=cern/OU=computers/CN=vocms105.cern.ch" node="gvocms105.cern.ch:9620-9719" secondary="True" group="2"/>
   </collectors>

Cheers,
Igor

PS: Sorry for commenting so late.

#4 Updated by Parag Mhashilkar almost 8 years ago

The example you described above is also covered in my proposal. Even if you don't add group the way it works with my proposal, the glidein will select one collector host from glidein-collector.t2.ucsd.edu:9620-9919 and other from gvocms105.cern.ch:9620-9719

The way it works now, frontend will never pick up and add to params.cfg any primary collector if atleast one secondary collector is configured. So essentially if the primary collector is on one host while the secondary is on the other, glidein will only report back to the secondary collector. That's why adding every collector we want glidein to report to as a secondary collector works nicely with my proposal. Grouping the collectors is not really required for glidein to select from.

I am not against adding group, but also want to make sure if we have a bug and the current behavior is unintended.

#5 Updated by Parag Mhashilkar almost 8 years ago

Rereading your comment, is this the use case you have in mind?

  <collectors groups="1,2">
      <collector DN="/DC=org/DC=doegrids/OU=Services/CN=glidein-collector.t2.ucsd.edu" node="glidein-collector.t2.ucsd.edu" secondary="False" group="1"/>
      <collector DN="/DC=org/DC=doegrids/OU=Services/CN=glidein-collector.t2.ucsd.edu" node="glidein-collector.t2.ucsd.edu:9620-9919" secondary="True" group="1"/>
      <collector DN="/DC=org/DC=doegrids/OU=Services/CN=glidein-collector.t3.ucsd.edu" node="glidein-collector.t3.ucsd.edu:9620-9919" secondary="True" group="1"/>
      <collector DN="/DC=ch/DC=cern/OU=computers/CN=vocms105.cern.ch" node="vocms105.cern.ch" secondary="False" group="2"/>
      <collector DN="/DC=ch/DC=cern/OU=computers/CN=vocms105.cern.ch" node="gvocms105.cern.ch:9620-9719" secondary="True" group="2"/>
      <collector DN="/DC=ch/DC=cern/OU=computers/CN=vocms106.cern.ch" node="gvocms106.cern.ch:9620-9719" secondary="True" group="2"/>
   </collectors>

So essentially, you want glidein to send adds to one of the {glidein-collector.t2.ucsd.edu:9620-9919; glidein-collector.t3.ucsd.edu:9620-9919} and another ad to one of the {gvocms105.cern.ch:9620-9719; gvocms106.cern.ch:9620-9719}

Agreed, in that case grouping is required as it is not clear to me if it will lead to double counting with my proposal.

#6 Updated by Igor Sfiligoi almost 8 years ago

Correct.

#7 Updated by Parag Mhashilkar almost 8 years ago

On a different note, the current config handling does strange hard coded checks for singular v/s plural to create either a list or dict of elements from the xml tag. Unless I am missing something there doesn't seem to be a way to add groups="xxx" in <collectors> tag without major code rewrite or breaking backward compatibility. Admin will have to delete/add for disabling/enabling groups.

#8 Updated by Igor Sfiligoi almost 8 years ago

OK.

The important part was having the group ID in the single collector lines;
then just create the internal "groups" set with the union of all of them.

When I have time, I will see if I can add support for the attributes in multiples.... but that's lower priority.

Thanks,
Igor

#9 Updated by Parag Mhashilkar almost 8 years ago

Igor Sfiligoi wrote:

When I have time, I will see if I can add support for the attributes in multiples.... but that's lower priority.

In a way there is support for attributes in multiples but for different type of xml tags, those with with singulars having name attributes. Tag collector is not of that type and will most likely break backward compatibility.

Other alternative is to extend base class list and use it for these multiples type.

But this can happen in future. I really doubt it will be useful for HA collector groups. It is not something that is likely to change often.

#10 Updated by Parag Mhashilkar almost 8 years ago

branch_v2plus commit:14871f6
head commit:7ccb1a6

#11 Updated by Parag Mhashilkar almost 8 years ago

  • Status changed from Assigned to Resolved

#12 Updated by Parag Mhashilkar over 7 years ago

  • Target version changed from v2_7_x to v2_6

#13 Updated by Parag Mhashilkar over 7 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF