Project

General

Profile

Feature #5345

Expand attribute expansion capabilities

Added by Igor Sfiligoi over 5 years ago. Updated 12 months ago.

Status:
Feedback
Priority:
Normal
Assignee:
Parag Mhashilkar
Category:
Configuration Management
Target version:
Start date:
02/04/2014
Due date:
% Done:

0%

Estimated time:
Stakeholders:

CMS, OSG

Duration:

Description

Right now, the only place where we allow for attribute expansion is in the FE expressions.

This functionality would instead be useful everywhere else.
Including nested/recursive (but no loops) expansion.


Related issues

Related to glideinWMS - Feature #2452: Access to config defined attributes in the match_exprClosed2012-02-06

History

#1 Updated by Igor Sfiligoi over 5 years ago

  • Priority changed from Normal to High

#2 Updated by Igor Sfiligoi over 5 years ago

Looking into the details of $$ expansion, I see it honors the attribute type.
Which means, strings get quoted during expansion.

This makes is useless for generic expansion;
e.g. could not be used to propagate the host name in the factory config.

Moreover, it is also hard to imagine doing recursive expansions this way.

We need a way to get "plain expansion" semantics.
I see three possible ways:
  1. just change the $$ semantics, and break backwards compatibility
  2. having different semantics depending where it is used, so current use cases are covered in backwards-compatible way
  3. use a different symbol for plain expansion, and keep $$ semantics as is (just extended to all strings)

I don't really like any of the above solutions, so if you can come up with a better idea, please let me know ASAP.

#3 Updated by Igor Sfiligoi over 5 years ago

Looking at the XML files I have access to, I see that single $ is only used in
GLIDEIN_CPUS=$(DETECTED_CORES)

I think we can deal with this backward incompatibility.

So I am proposing to use single $() expansion for straight replacement, and keep $$() for quoted replacement.

We also define $(DOLLAR) as being expanded to $.

If anyone objects to this plan, please let me know ASAP.

#4 Updated by Igor Sfiligoi over 5 years ago

Looking in more details, it looks like this is going to be harder than I thought, unless we make some semantic changes.

The main problem is that we may have entry/group attributes influence the global values.
Since we write (a subset of) the global values to disk, and reuse them for all the entries/groups,
we would need to do the expansion at runtime.
Which would include both the python processes AND in the glideins.
And in the glideins we actually currently lacking some of the info, too.

I don't think that is really reasonable, so I am proposing to "promote" any global value that needs expansion into the entry/group level.
This way I can do the expansion at (re)config time.

This may slightly increase the disc space used, but I think the benefits way outweigh the drawbacks.

Let me know if you disagree with this plan.

#5 Updated by Parag Mhashilkar over 5 years ago

I am in favor of putting everything in the leaf node (directory). This way the entry directory can be populated at reconfig time and will be functional by itself. Might as well simplify the code.

#6 Updated by Igor Sfiligoi over 5 years ago

  • Status changed from Assigned to Feedback
  • Assignee changed from Igor Sfiligoi to Parag Mhashilkar
  • Priority changed from High to Normal
  • Target version changed from v3_x to v3_2_x

The proposed semantics was implemented both for factory and frontend.
I have committed the code in
v3/5345

Please review.

PS: I am selectively promoting only attributes than need expansion.
As mentioned on the phone call, we can extend that in a separate ticket.

#7 Updated by Igor Sfiligoi over 5 years ago

  • Status changed from Feedback to Assigned
  • Assignee changed from Parag Mhashilkar to Igor Sfiligoi

Further testing showed some bugs.
Re-taking ownership of it.

#8 Updated by Igor Sfiligoi over 5 years ago

  • Status changed from Assigned to Feedback
  • Assignee changed from Igor Sfiligoi to Parag Mhashilkar

Fixed the discovered bugs (still in v3/5345).

Should be ready to be reviewed.

#9 Updated by Igor Sfiligoi over 5 years ago

  • Target version changed from v3_2_x to v3_2_6

Merged with v3_2_4 and fixed the conflicts.

Now in v3/5345_v2.

#10 Updated by Parag Mhashilkar about 5 years ago

  • Assignee changed from Parag Mhashilkar to Igor Sfiligoi

Reviewed v3/5345_v2 and sent feedback separately.

#11 Updated by Burt Holzman about 5 years ago

  • Target version changed from v3_2_6 to v3_2_x

#12 Updated by Igor Sfiligoi about 5 years ago

  • Assignee changed from Igor Sfiligoi to Parag Mhashilkar
  • Target version changed from v3_2_x to v3_2_7

I think I have addressed all the points, including adding the documentation.

The new code is now in v3/5345_v3.

Please review.

#13 Updated by Burt Holzman almost 5 years ago

  • Target version changed from v3_2_7 to v3_2_8

#14 Updated by Parag Mhashilkar over 4 years ago

  • Target version changed from v3_2_8 to v3_2_x

#15 Updated by Marco Mambelli over 1 year ago

  • Target version changed from v3_2_x to v3_4_x

#16 Updated by Marco Mambelli 12 months ago

  • Target version changed from v3_4_x to v3_5_x


Also available in: Atom PDF