Feature #14660: POMS should actually hold job launches when services are "red" on dashboard. - Production Operations Management Service (POMS) - Fermilab Redmine

Feature #14660

POMS should actually hold job launches when services are "red" on dashboard.

Added by Marc Mengel about 3 years ago. Updated about 3 years ago.

Status:

Closed

Priority:

Normal

Assignee:

Target version:

v1_1_0

Start date:

11/24/2016

Due date:

11/24/2016

% Done:

100%

Estimated time:

Scope:

Internal

Experiment:

Stakeholders:

Duration: 1

Description

POMS should actually hold job launches when services are "red" on dashboard.

Related issues

Associated revisions

Revision 1e59c4d7 (diff)
Added by Marc Mengel about 3 years ago

rough draft of issue #14660

History

#1 Updated by Marc Mengel about 3 years ago

Target version set to v1_1_0

#2 Updated by Marc Mengel about 3 years ago

Follows Feature #14659: Poms should keep a list of job launches requested while job launches are "held" added

#3 Updated by Marc Mengel about 3 years ago

Coded this up in 1e59c4d when we start/end downtimes on the "All" item, we will now hold/unhold jobs.

We should probably also send mail(?) or something.

Also updated the configs for several of our monitoring items to show degraded sooner and failed less often, so this won't stall us out too often.

Actually, now that I look at it, the way we hold jobs is probably unhappy in the uwsgi universe; we had a shared config object in the cherrypy configuraion, but I don't think we do in uwsgi; so modifying it won't get to the other uwsgi processes. Probably needs a single-row table... or we could stash it in Services...