Project

General

Profile

Feature #3532

Propagate to Factory and glidein submission attributes controlled by FE

Added by Anthony Tiradani over 6 years ago. Updated 5 months ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Start date:
02/26/2013
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Stakeholders:

HEPCloud

Duration:

Description

I am getting requests to handle cloud features like spot pricing and availability zones. HTCondor already handles these features, but we don't.

The two examples I listed are attributes that need to show up in the submit file. However, these attributes need to be controlled by the Frontend admins and not the Factory admins. (Do you really want to reconfigure the factory every time a FE admin asks for a spot price change?)

These attributes are optional. So, if a spot price is specified, a spot instance is requested. If the spot price is omitted (not specified), a normal instance is requested. If an availability zone is requested, the request will be made for that zone, if it is omitted, the zone defaults to "default".

So I see two improvements that we need to make. (I will make child tickets for each.)

1. We need to find a way to pass the FE controlled values to the Factory. We currently shoe-horn in the vm id and the vm type into the credentials, but this isn't ideal.

2. We need to handle optional attributes at submit time.


Subtasks

Feature #3534: We need to find a way to pass the FE controlled values to the Factory.ClosedMarco Mambelli

Feature #3537: We need to handle optional attributes at submit timeClosed

History

#1 Updated by Igor Sfiligoi over 6 years ago

Hi Tony.

Let' make sure we don't mix things.

You are of course right... the spot pricing is something the FE should be setting... not a factory business.
(Something the factory could do is instead advertising the lowest spot price possible and the maybe the current and avg one)

Availability zones are instead a "entry property"... we should have treat each availability zone as its own "queue". I don't think this should be any different than what we do in the Grid world.

Bottom line, I think we will be doing the VO FE admins a big disservice if we require them to micromanage the resources.
(See the Coral discussion on how to handle big HPC resources for more details)

Igor

#2 Updated by Igor Sfiligoi over 6 years ago

Now, a concrete proposal for the spot pricing.

We could easily handle it the same way we handle the "project id" in Teragrid.

To me, they are very similar, conceptually. Both add a constraint on the resources requested to the service provider.

#3 Updated by Burt Holzman over 6 years ago

  • Assignee set to Burt Holzman

#4 Updated by Burt Holzman over 6 years ago

  • Target version set to v3_1

#5 Updated by Burt Holzman over 6 years ago

  • Target version changed from v3_1 to v3_x

#6 Updated by Marco Mambelli 8 months ago

  • Assignee changed from Burt Holzman to Marco Mambelli
  • Target version changed from v3_x to v3_5_1

#7 Updated by Marco Mambelli 7 months ago

  • Target version changed from v3_5_1 to v3_5
  • Assignee changed from Marco Mambelli to Lorena Lobato Pardavila

Add to the submit environment attributes that are published to the classad.
Use a prefix "GLIDEIN_PARAM_..."

Changes can probably be in glideFactoryLib.py, in:

def get_submit_environment(entry_name, client_name, submit_credentials,
                           client_web, params, idle_lifetime, log=logSupport.log,
                           factoryConfig=None):

#8 Updated by Lorena Lobato Pardavila 6 months ago

  • Status changed from New to Work in progress

#9 Updated by Lorena Lobato Pardavila 6 months ago

  • Assignee changed from Lorena Lobato Pardavila to Dennis Box
  • Status changed from Work in progress to Feedback

Changes done in v35/3532

Now, to pass the Frontend controlled values that affect Factory submission, needed to set the attribute parameter="True" in the frontend.xml.

Note for the future: The attribute name "publish" is not available for the Frontend for the moment. But this is something that we'll add to the TODO list as we would like to have some control over the length of the parameters list that is passed to the submission of the glidein.

#10 Updated by Dennis Box 6 months ago

  • Assignee changed from Dennis Box to Lorena Lobato Pardavila
  • Status changed from Feedback to Accepted

feedback sent

#11 Updated by Marco Mambelli 6 months ago

  • Target version changed from v3_5 to v3_4_4

#12 Updated by Lorena Lobato Pardavila 6 months ago

  • Assignee changed from Lorena Lobato Pardavila to Dennis Box
  • Status changed from Accepted to Feedback

#13 Updated by Marco Mambelli 6 months ago

  • Assignee changed from Dennis Box to Lorena Lobato Pardavila
  • Subject changed from We need to handle attributes controlled by FE that affects factory submission to Propagate to Factory and glidein submission attributes controlled by FE

Title changed from "We need to handle attributes controlled by FE that affects factory submission" to "Propagate to Factory and glidein submission attributes controlled by FE".

The initial request was to:
1. We need to find a way to pass the FE controlled values to the Factory. We currently shoe-horn in the vm id and the vm type into the credentials, but this isn't ideal.
2. We need to handle optional attributes at submit time.

This is obtained by the ticket in the following way:
1. is done by the ticket by publishing in the glideclient classad and in the glidein submit environment when parameter="True" in the attr specification in the Frontend config.
2. is done by accessing the unix environment in the Glidein JDL (using $ENV )

NOTE for the developer:
- I think a prefix added to the attr name when publishing it in the submit environment. This is good because because a wrong attribute could override important unix variables (PATH, ...) and break the submission in the factory.

This should be highlighted in the documentation. Otherwise, people may expect to use the same name in the JDL.

#14 Updated by Lorena Lobato Pardavila 6 months ago

  • Status changed from Feedback to Resolved

Merged into master after feedback.

For the record, and as highlighted in the documentation, the FE controlled values to the Factory will be published in the submit environment with the prefix "GLIDEIN_PARAM_".

#15 Updated by Marco Mambelli 6 months ago

  • Stakeholders updated (diff)

#16 Updated by Marco Mambelli 5 months ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF