Allow a DPM to have a custom configuration
Jim Smedinghoff reports that, while preparing for this morning's power outage, the new DPM on
CLX5 had problems and needed to be restarted. We weren't able to investigate the DPM before it was restarted, so this is an (educated) guess:
- Before the power outage, Jim shutdown each DPM outside of the control room. The one running on
CLX5was left alone since it's on a UPS.
- As each DPM was brought down, all associated clients rerouted their requests to any remaining DPMs.
- Eventually, the only remaining DPM was the one on
We know one DPM isn't enough to handle all the Control System's clients, so my guess is that
CLX5 had high memory and CPU usage and put it an unusable state. Even though most of its clients went away during the power outage, the damage was already done (high resource usage). Maybe it would have eventually recovered, but it was quicker to just restart the one instance.
My recommendation is to make the
CLX5 DPM only accept requests from Main Control Room consoles.
Erlang virtual machines have an associated configuration file which is specified when the VM is started. Right now we're using the default name
sys.config. On our development nodes, we can edit this file to tweak configuration variables. One of these variables indicates which nodes from which the DPM will accepts requests and, by default, we set it up to accept requests from every node.
When we build a DPM release, all DPMs get the same
sys.config, so clearly we have to do something to make
CLX5 unique. However, I don't want to implement this feature directly in the DPM source.
What I propose is we include multiple
.config files in the release, including the default
sys.config. The script that starts DPM can check to see if there's a configuration for the node it's on and use that instead of the default. For instance, when DPM is started on
CLX5, the script will look for
clx5.config. If that doesn't exist, then
sys.config is used.
#4 Updated by Richard Neswold almost 3 years ago
- Status changed from New to Closed
- Assignee set to Richard Neswold
Rather than mess with shell scripts and multiple config files, Jim Smedinghoff suggested that the config format could be changed to handle multiple DPMs. The commit shows how simple it was using this suggestion. In commit c71f7e81, I set up
CLX5 to only accept connections from consoles in the MCR.
This code will get released in v1.7.