Project

General

Profile

Feature #3206

Feature #3203: OSG: condor_q -analyze analogue for glideins

Expose Frontend policy in the ClassAds

Added by Igor Sfiligoi over 7 years ago. Updated about 7 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Douglas Strain
Category:
-
Target version:
Start date:
12/21/2012
Due date:
% Done:

0%

Estimated time:
Stakeholders:
Duration:

Description

Currently the matching policy of the frontend is only known to the Frontend itself. This makes it impossible to write a tool that would check if a job matches an entry.

The ideal candidate for this seem to be the glideresource classads, although one should double check.

Also, ideally we would expose both the FE matching policy itself, and the policy the glideins will use when started (at least, the FE-provided part).

History

#1 Updated by Burt Holzman over 7 years ago

  • Assignee changed from Burt Holzman to Douglas Strain

#2 Updated by Burt Holzman over 7 years ago

  • Status changed from New to Assigned

#3 Updated by Douglas Strain over 7 years ago

  • Status changed from Assigned to Feedback
  • Assignee changed from Douglas Strain to Parag Mhashilkar

This has been done in branch_v2plus_3206 (commit:e7e5cd3):

I have added four attributes to the glideresource classad. I think I have gotten all the various expressions of interest:
- GlideClientMatchExpr: frontend's (merged) <match match_expr>
- GlideClientStartExpr frontend's <match start_expr>
- GlideClientJobQueryExpr: frontend's (merged) <factory query_expr>
- GlideClientFactoryQueryExpr: frontend's (merged) <job query_exp>

An example is below:

  1. condor_status -any -constraint "MyType == \"glideresource\"" -l | grep Expr
    GlideClientStartExpr = "(stringListMember(GLIDEIN_Site,DESIRED_Sites,\",\")=?=True)"
    GlideClientJobQueryExpr = "((JobUniverse==5)&&(GLIDEIN_Is_Monitor =!= TRUE)&&(JOB_Is_Monitor =!= TRUE)) && (True)"
    GlideClientMatchExpr = "((job.get(\"DESIRED_Sites\")!=None) and glidein[\"attrs\"].get(\"GLIDEIN_Site\") in (job.get(\"DESIRED_Sites\").split(\",\"))) and (True)"
    GlideClientFactoryQueryExpr = "((GLIDEIN_Site=!=UNDEFINED)) && (True)"

Please review and let me know any comments.

#4 Updated by Parag Mhashilkar over 7 years ago

  • Assignee changed from Parag Mhashilkar to Douglas Strain

Looks ok to merge. Please also merge it in master

#5 Updated by Igor Sfiligoi over 7 years ago

I have a comment:
Should we define the types of the above?

Some are ClassAd expressions, while others are Python expressions.

As long as it is all in the frontend XML is one thing, but the ClassAd may be parsed by external tools.

#6 Updated by Igor Sfiligoi over 7 years ago

Also, we need to document this.

Doug's commit is only about code changes.

#7 Updated by Burt Holzman over 7 years ago

We discussed the naming of *Expr back in #3125. Glad you're coming around.

Doug, what do you think of renaming all but the MatchExpr to *CondorExpr instead?

- B

#8 Updated by Douglas Strain over 7 years ago

I agree, I should document this.

Igor, what did you mean by defining the types?
Did you mean changing "GlideClientMatchExpr" to "GlideClientMatchPythonExpr" for instance?
Or is there a condor trick for defining the types in the classad itself?

As Burt just suggested, I am okay with renaming things to be more descriptive.

#9 Updated by Parag Mhashilkar over 7 years ago

We can rename the attrs to following. Does this work?

GlideClientCondorStartExpr = "(stringListMember(GLIDEIN_Site,DESIRED_Sites,\",\")=?=True)"
GlideClientCondorJobQueryExpr = "((JobUniverse==5)&&(GLIDEIN_Is_Monitor =!= TRUE)&&(JOB_Is_Monitor =!= TRUE)) && (True)"
GlideClientPythonMatchExpr = "((job.get(\"DESIRED_Sites\")!=None) and glidein[\"attrs\"].get(\"GLIDEIN_Site\") in (job.get(\"DESIRED_Sites\").split(\",\"))) and (True)"
GlideClientCondorFactoryQueryExpr = "((GLIDEIN_Site=!=UNDEFINED)) && (True)"

Igor Sfiligoi wrote:

I have a comment:
Should we define the types of the above?

Some are ClassAd expressions, while others are Python expressions.

As long as it is all in the frontend XML is one thing, but the ClassAd may be parsed by external tools.

#10 Updated by Igor Sfiligoi over 7 years ago

How about:
GlideClientStartCondorExpr = "(stringListMember(GLIDEIN_Site,DESIRED_Sites,\",\")=?=True)"
GlideClientJobQueryCondorExpr = "((JobUniverse==5)&&(GLIDEIN_Is_Monitor =!= TRUE)&&(JOB_Is_Monitor =!= TRUE)) && (True)"
GlideClientMatchPythonExpr = "((job.get(\"DESIRED_Sites\")!=None) and glidein[\"attrs\"].get(\"GLIDEIN_Site\") in (job.get(\"DESIRED_Sites\").split(\",\"))) and (True)"
GlideClientFactoryQueryCondorExpr = "((GLIDEIN_Site=!=UNDEFINED)) && (True)"

?

#11 Updated by Douglas Strain over 7 years ago

I like Igor's naming scheme better. I'll go ahead and change that and add documentation unless anyone objects.

#12 Updated by Igor Sfiligoi over 7 years ago

Actually...
one more comment.

Why are you prefixing it with
GlideClient
?

If I remember correctly, GlideClient prefix indicates attributes we are sending to the Factory.

No concrete counter-proposal yet... let me think a bit about it.

#13 Updated by Parag Mhashilkar over 7 years ago

It depends on what you want to associate the type with, Expr or StartExpr. I am fine with either StartCondorExpr or CondorStartExpr. Similarly for other 3 variables.

What ever we decide we should be consistent wrt xml config.

#14 Updated by Douglas Strain over 7 years ago

I prefixed it GlideClient since it is coming from the frontend (client). I guess the eventual use is that the factory may want to know this for debugging reasons, so we are sort of sending it to the factory.

That being said, I am not opposed to changing it. I am not sure if we have anything else similar in that classad. Maybe GlideFrontend or GlideInfo? I think I like GlideClient better than those though. Any better suggestions?

#15 Updated by Igor Sfiligoi over 7 years ago

No, you are right.
GlideClient is the right prefix.

What is/was bothering me is that there is no grouping between them.

So, here is my proposal; I know is a radical change, but seems more logical to me.
(and, no, it is not following the XML naming at all)

I would extend the prefix to
GlideClientMatching
and
GlideClientConstraint
?

And then rename the attributes to
GlideClientMatchingInternalPythonExpr = "((job.get(\"DESIRED_Sites\")!=None) and glidein[\"attrs\"].get(\"GLIDEIN_Site\") in (job.get(\"DESIRED_Sites\").split(\",\"))) and (True)"
GlideClientMatchingGlideinCondorExpr = "(stringListMember(GLIDEIN_Site,DESIRED_Sites,\",\")=?=True)"
GlideClientConstraintJobCondorExpr = "((JobUniverse==5)&&(GLIDEIN_Is_Monitor =!= TRUE)&&(JOB_Is_Monitor =!= TRUE)) && (True)"
GlideClientConstraintFactoryCondorExpr = "((GLIDEIN_Site=!=UNDEFINED)) && (True)"

Igor

Douglas Strain wrote:

I prefixed it GlideClient since it is coming from the frontend (client). I guess the eventual use is that the factory may want to know this for debugging reasons, so we are sort of sending it to the factory.

That being said, I am not opposed to changing it. I am not sure if we have anything else similar in that classad. Maybe GlideFrontend or GlideInfo? I think I like GlideClient better than those though. Any better suggestions?

#16 Updated by Douglas Strain over 7 years ago

Ok, I have done the documentation for the change to the classad (commit b2a88a4).

I'll wait until after the holiday weekend to get consensus on the variable names.

#17 Updated by Parag Mhashilkar over 7 years ago

  • Target version changed from v2_7_x to v2_7

Looks ok to merge

#18 Updated by Douglas Strain over 7 years ago

  • Status changed from Feedback to Resolved

This change has been merged into branch_v2plus and master

#19 Updated by Igor Sfiligoi over 7 years ago

  • Status changed from Resolved to Feedback

Wait a second...

Weren't we suppose to agree on a naming convention?

What was committed?

#20 Updated by Douglas Strain over 7 years ago

Sorry, I should have been more explicit in my update. We discussed during the weekly meeting and decided that your final suggestion was the most descriptive. Hence, we have chosen this naming scheme:

GlideClientMatchingInternalPythonExpr = "((job.get(\"DESIRED_Sites\")!=None) and glidein[\"attrs\"].get(\"GLIDEIN_Site\") in (job.get(\"DESIRED_Sites\").split(\",\"))) and (True)"
GlideClientMatchingGlideinCondorExpr = "(stringListMember(GLIDEIN_Site,DESIRED_Sites,\",\")=?=True)"
GlideClientConstraintJobCondorExpr = "((JobUniverse==5)&&(GLIDEIN_Is_Monitor =!= TRUE)&&(JOB_Is_Monitor =!= TRUE)) && (True)"
GlideClientConstraintFactoryCondorExpr = "((GLIDEIN_Site=!=UNDEFINED)) && (True)"

#21 Updated by Igor Sfiligoi over 7 years ago

  • Status changed from Feedback to Resolved

Thanks

#22 Updated by Parag Mhashilkar about 7 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF