Feature #20766

Change configuration handling and configuration files format

Added by Marco Mambelli over 2 years ago. Updated 9 months ago.

Configuration Management
Target version:
Start date:
Due date:
% Done:


Estimated time:


Factory and Frontend configuration files are in XML
Furthermore they are being modified sometimes during the reconfig/upgrade commands

The new configuration files
- should be easy to write/read
- should allow comments
- should allow inclusion of multiple files
- should never be modified by reload(reconfig) or upgrade or restart

There should be a tool for checking/completing the configuration (different form the reloading). This could modify the file in place or write a different version.

Variables lists like condor_vars.lst should always be used to check correct type/options of the attributes and to set default values (see [#20749])

Some considerations based on current configuration handling in GWMS:
- should be in a different format from XML, easier to write and check (maybe Python, or JSON derivative like YAML)
- Upgrade could become an option of restart (or always be done at restart)
- Reloading should always happen on restart. And remain an option during run-time

Generic considerations:
- operators should evaluate the formats (e.g. presenting mock-ups in different formats) and pick one
- better if it allows defaults specification (e.g. for all attributes the dafault is... following attributes lists will include only elements that change)
- better if it allows type annotation and verification (e.g. enumerates)


#1 Updated by Marco Mambelli over 2 years ago

Added consideration:
- the configuration should allow the execution of code to generate some content (dynamic content/plugins)
- the configuration framework should be the same for Factory and Frontend, keeping as much as possible common (parsing, conversion, default managements, plugins, framework) and clearly isolate parts that are different (specific constraint and content)

Formats to consider:

#2 Updated by Marco Mambelli over 1 year ago

  • Assignee set to Fernando Lisboa

#4 Updated by Marco Mambelli over 1 year ago

Another format that could be considered is FHiCL. It has some interesting extensions in variables handling and allows imports:
- developed and used only at Fermilab (limited use, limited language support, anyway C++ and Python are supported)
+ implicit references: all elements can be referenced using dot notation and seq index) (note the sequential evaluation of references, different from HTCondor)
- everything can be a variable (this can be confusing in long files)
+ binding protection (ignore reassignment or throw error, a way to have constants),
+ prolog sections where default values could be specified
+ native import support (in YAML is possible w/ extensions not in the spec
- single-line comments
- unclear multi-line string support
- only basic (json) data types (no sets, ordered dicts, ...)
- no type annotation

This should be compared to YAML

#5 Updated by Marco Mambelli over 1 year ago

Fernando did a Python-YAML comparison with a mocked Factory config and a parser. The results are in github:

Check the testing branch:

#6 Updated by Marco Mambelli over 1 year ago

  • Description updated (diff)

#7 Updated by Marco Mambelli over 1 year ago

Adding some new information:

Would be nice to have something similar for YAML

Ansible has some extensions to YAML:

"{{ variable }}" for variables

YAML spec special chars:

- ? :  list item, key, value
, [] {}  for flow notation (c-flow-indicator)
& * !  anchor, alias, tag
# comment
| > flow and folded block
' "  single and double quoted strings
% directive line
` @ reserved for future use
cr lf are the only line-break characters
sp tab are the only white-space characters

PyYAML allows you to attach custom constructors (such as !include) to the YAML loader

import yaml
import os

class Loader(yaml.SafeLoader):

    def __init__(self, stream):

        self._root = os.path.split([0]

        super(Loader, self).__init__(stream)

    def include(self, node):

        filename = os.path.join(self._root, self.construct_scalar(node))

        with open(filename, 'r') as f:
            return yaml.load(f, Loader)

Loader.add_constructor('!include', Loader.include)

Here a gist with a better python3 implementation and usage example:

Symphony is a YAML library for PHP that has different extensions:

Hiera - yaml hierarchy from puppet

Also available in: Atom PDF