Project

General

Profile

Task #14029

Monitoring: Make a Google Gratia probe

Added by Steven Timm almost 3 years ago. Updated about 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Start date:
10/04/2016
Due date:
11/14/2016
% Done:

10%

Estimated time:
8.00 h
Duration: 42

Description

HAve to decide, and modify the AWSVM probe if necesary, shouldn't be too different.

History

#1 Updated by Steven Timm over 2 years ago

  • Status changed from New to Remission

Won't start work on this until there is another approved demo target.

#2 Updated by Steven Timm over 2 years ago

  • Subject changed from Do we need a google Gratia probe to Make a Google Gratia probe
  • Status changed from Remission to New

#3 Updated by Steven Timm over 2 years ago

  • Due date set to 11/14/2016
  • Estimated time set to 8.00 h

Can use a lot of the same code as was used in the gratia-probe-awsvm

#4 Updated by Steven Timm over 2 years ago

  • Subject changed from Make a Google Gratia probe to Monitoring: Make a Google Gratia probe
  • Assignee set to Joe Boyd

#5 Updated by Joe Boyd over 2 years ago

Background:

For AWS, the gratia-probe-awsvm Gratia probe and the monitoring code for FIFEmon
were almost the same

code written by the same student last summer, Shivakumar Vinayagam.

For Google cloud we have monitoring code written by a summer student but no
gratia probe code yet,

but all the calls you need for the Gratia probe are in that monitoring code.

A major difference between Google and AWS is that the price does not change on
an hour-by-hour basis, there

is one fixed price for the regular instances and a different fixed price for the
so-called "preemptible" instances which can be pre-empted at any time. Another
major difference is that after the first 10 minutes, Google bills in minute
increments. A third major difference is that instances made by the glideinwms
factory are deleted from

the compute instance list immediately when they are terminated, instead of
hanging around for another hour.

Monitoring code we have is in redmine git repository "monitoringaccountingbilling"

you can fetch it by

git clone
ssh://:/cvs/projects/monitoringaccountingbililng

the interesting stuff is under trunk/src/monitor-gce/bin

Since the code is of similar structure to the gratia-probe-awsvm it should not
be too hard to merge in

the appropriate routines.

You will need to install python 2.7 and the google sdk on any machine you are
testing it on, to make it work.

The Google API calls you will need are described in

https://cloud.google.com/compute/docs/reference/latest/

API Reference | Compute Engine | Google Cloud Platform
<https://cloud.google.com/compute/docs/reference/latest/>
cloud.google.com
Method HTTP request Description; URIs relative to
https://www.googleapis.com/compute/v1/projects, unless otherwise noted: delete:
DELETE /project/global ...

and the monitoring (Stackdriver) calls you will need are described here:

https://cloud.google.com/monitoring/api/ref_v3/rest/

Stackdriver Monitoring API - Google Cloud Platform
<https://cloud.google.com/monitoring/api/ref_v3/rest/>
cloud.google.com
Manages your Stackdriver Monitoring data and configurations. Most projects must
be associated with a Stackdriver account, with a few exceptions as noted on the ...

Steve Timm

#6 Updated by Joe Boyd over 2 years ago

Looks like the AWS probe populates these gratia UsageRecord fields:

r = Gratia.UsageRecord()
r.LocalUserId(user)
r.GlobalUsername(owneracct)
r.ProjectName(project)
r.VOName(voname)
r.ReportableVOName(voname)
r.LocalJobId(instance['InstanceId'])
r.GlobalJobId(instance['InstanceId']+"#"+repr(time.time()))
r.JobName(tag['Value'])
r.Status(status,description)
#r.ProcessorsDescription(instance['InstanceType'])
#r.MachineNameDescription(instance['ImageId'])
r.ResourceType('aws')
r.Njobs(1,"The no of jobs running at a time")
r.NodeCount(1) # default to total
r.Processors(int(cpu),0,"total",instance['InstanceType'])
r.Memory(float(memory))
r.Charge(str(price),"$","$/instance hr",chargedesc)
r.StartTime(stime)
r.EndTime(etime)
r.WallDuration(etime-stime)
r.CpuDuration(0,'user')
r.CpuDuration(0,'system')
r.ResourceType("AWSVM")
r.AdditionalInfo("Version","1.0")
Gratia.Send(r)

Will try to get at least those from GCE and anything else that can be filled into the UsageRecord.

#7 Updated by Steven Timm over 2 years ago

I will try to find the document in which Shiva defined how he populated those fields.
We should have it.

#8 Updated by Joe Boyd over 2 years ago

The code is simple so I think I'm good. I'm just noting the info we logged from AWS so I'll get at least that much populated from the GCE API. If you have any writeups/docs though I'd be happy to take a look.

#9 Updated by Steven Timm over 2 years ago

  • Status changed from New to Work in progress
  • % Done changed from 0 to 10

As of this morning, Joe says this is not ready to test yet.

#10 Updated by Steven Timm about 1 year ago

  • Project changed from Google Cloud Testing to AdministrationOfCloudServices
  • Assignee deleted (Joe Boyd)

#11 Updated by Steven Timm about 1 year ago

  • Status changed from Work in progress to Resolved

Will track this in gitlabs

#12 Updated by Steven Timm about 1 year ago

  • Status changed from Resolved to Closed

Will track this in gitlabs



Also available in: Atom PDF