Project

General

Profile

Feature #16522

Allow a DPM to have a custom configuration

Added by Richard Neswold over 3 years ago. Updated almost 3 years ago.

Status:
Closed
Priority:
Normal
Category:
Data Pool Manager
Target version:
Start date:
05/15/2017
Due date:
% Done:

0%

Estimated time:
Duration:

Description

Jim Smedinghoff reports that, while preparing for this morning's power outage, the new DPM on CLX5 had problems and needed to be restarted. We weren't able to investigate the DPM before it was restarted, so this is an (educated) guess:

  • Before the power outage, Jim shutdown each DPM outside of the control room. The one running on CLX5 was left alone since it's on a UPS.
  • As each DPM was brought down, all associated clients rerouted their requests to any remaining DPMs.
  • Eventually, the only remaining DPM was the one on CLX5.

We know one DPM isn't enough to handle all the Control System's clients, so my guess is that CLX5 had high memory and CPU usage and put it an unusable state. Even though most of its clients went away during the power outage, the damage was already done (high resource usage). Maybe it would have eventually recovered, but it was quicker to just restart the one instance.

My recommendation is to make the CLX5 DPM only accept requests from Main Control Room consoles.

Implementation

Erlang virtual machines have an associated configuration file which is specified when the VM is started. Right now we're using the default name sys.config. On our development nodes, we can edit this file to tweak configuration variables. One of these variables indicates which nodes from which the DPM will accepts requests and, by default, we set it up to accept requests from every node.

When we build a DPM release, all DPMs get the same sys.config, so clearly we have to do something to make CLX5 unique. However, I don't want to implement this feature directly in the DPM source.

What I propose is we include multiple .config files in the release, including the default sys.config. The script that starts DPM can check to see if there's a configuration for the node it's on and use that instead of the default. For instance, when DPM is started on CLX5, the script will look for clx5.config. If that doesn't exist, then sys.config is used.

History

#1 Updated by Richard Neswold over 3 years ago

  • Description updated (diff)

I guess I should have better described how the feature would be implemented.

#2 Updated by Richard Neswold about 3 years ago

  • Category set to Data Pool Manager
  • Target version set to DPM v1.6

#3 Updated by Richard Neswold about 3 years ago

  • Target version changed from DPM v1.6 to DPM v1.7

Move to release v1.7 so we don't delay some important bug fixes found in v1.6.

#4 Updated by Richard Neswold almost 3 years ago

  • Status changed from New to Closed
  • Assignee set to Richard Neswold

Done: 907eb955

Rather than mess with shell scripts and multiple config files, Jim Smedinghoff suggested that the config format could be changed to handle multiple DPMs. The commit shows how simple it was using this suggestion. In commit c71f7e81, I set up CLX5 to only accept connections from consoles in the MCR.

This code will get released in v1.7.

Also available in: Atom PDF