Project

General

Profile

Feature #14660

POMS should actually hold job launches when services are "red" on dashboard.

Added by Marc Mengel about 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
Start date:
11/24/2016
Due date:
11/24/2016
% Done:

100%

Estimated time:
Scope:
Internal
Experiment:
-
Stakeholders:
Duration: 1

Description

POMS should actually hold job launches when services are "red" on dashboard.


Related issues

Follows Production Operations Management Service (POMS) - Feature #14659: Poms should keep a list of job launches requested while job launches are "held"Closed11/23/2016

Associated revisions

Revision 1e59c4d7 (diff)
Added by Marc Mengel almost 3 years ago

rough draft of issue #14660

History

#1 Updated by Marc Mengel about 3 years ago

  • Target version set to v1_1_0

#2 Updated by Marc Mengel about 3 years ago

  • Follows Feature #14659: Poms should keep a list of job launches requested while job launches are "held" added

#3 Updated by Marc Mengel almost 3 years ago

Coded this up in 1e59c4d when we start/end downtimes on the "All" item, we will now hold/unhold jobs.

We should probably also send mail(?) or something.

Also updated the configs for several of our monitoring items to show degraded sooner and failed less often, so this won't stall us out too often.

Actually, now that I look at it, the way we hold jobs is probably unhappy in the uwsgi universe; we had a shared config object in the cherrypy configuraion, but I don't think we do in uwsgi; so modifying it won't get to the other uwsgi processes. Probably needs a single-row table... or we could stash it in Services...

#4 Updated by Marc Mengel almost 3 years ago

Okay, so now we're storing our job launch status in a "job_launches" service in the Services table.

#5 Updated by Marc Mengel almost 3 years ago

  • % Done changed from 0 to 90

#6 Updated by Marc Mengel almost 3 years ago

  • Status changed from New to Resolved
  • % Done changed from 90 to 100

#7 Updated by Marc Mengel almost 3 years ago

Pre-populated production with the job_launches service already...

#8 Updated by Marc Mengel almost 3 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF