Feature #20766
Change configuration handling and configuration files format
0%
Description
Factory and Frontend configuration files are in XML
Furthermore they are being modified sometimes during the reconfig/upgrade commands
The new configuration files
- should be easy to write/read
- should allow comments
- should allow inclusion of multiple files
- should never be modified by reload(reconfig) or upgrade or restart
There should be a tool for checking/completing the configuration (different form the reloading). This could modify the file in place or write a different version.
Variables lists like condor_vars.lst should always be used to check correct type/options of the attributes and to set default values (see [#20749])
Some considerations based on current configuration handling in GWMS:
- should be in a different format from XML, easier to write and check (maybe Python, or JSON derivative like YAML)
https://www.zionandzion.com/json-vs-xml-vs-toml-vs-cson-vs-yaml/
- Upgrade could become an option of restart (or always be done at restart)
- Reloading should always happen on restart. And remain an option during run-time
Generic considerations:
- operators should evaluate the formats (e.g. presenting mock-ups in different formats) and pick one
- better if it allows defaults specification (e.g. for all attributes the dafault is... following attributes lists will include only elements that change)
- better if it allows type annotation and verification (e.g. enumerates)
History
#1 Updated by Marco Mambelli over 2 years ago
Added consideration:
- the configuration should allow the execution of code to generate some content (dynamic content/plugins)
- the configuration framework should be the same for Factory and Frontend, keeping as much as possible common (parsing, conversion, default managements, plugins, framework) and clearly isolate parts that are different (specific constraint and content)
Formats to consider:
Python, YAML, HJSON, JSON5?
#2 Updated by Marco Mambelli over 1 year ago
- Assignee set to Fernando Lisboa
#3 Updated by Marco Mambelli over 1 year ago
Here some links I found about configuration files formats:
https://stackoverflow.com/questions/1773805/how-can-i-parse-a-yaml-file-in-python
https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats
https://martin-thoma.com/configuration-files-in-python/
https://stackoverflow.com/questions/791761/xml-for-configuration-files-why
https://yaml.org/spec/1.2/spec.html#id2759732
Including an interesting library:
https://github.com/MartinThoma/cfg_load
Some things to keep in mind about YAML:
- spec v1.2 comes in 3 flavors (Failsafe, JSON and Core schema) and is more strict of 1.1 (especially the JSON schema) that is a superset of JSON
- use true, false, null (empty scalar may be an empty string)
https://perlpunk.github.io/slides.tpcig2018/yamlpp/slide021.html
http://blogs.perl.org/users/tinita/2018/01/introduction-to-yaml-schemas-and-tags.html
https://yaml.readthedocs.io/en/latest/pyyaml.html
#4 Updated by Marco Mambelli over 1 year ago
Another format that could be considered is FHiCL. It has some interesting extensions in variables handling and allows imports:
- developed and used only at Fermilab (limited use, limited language support, anyway C++ and Python are supported)
+ implicit references: all elements can be referenced using dot notation and seq index) (note the sequential evaluation of references, different from HTCondor)
- everything can be a variable (this can be confusing in long files)
+ binding protection (ignore reassignment or throw error, a way to have constants),
+ prolog sections where default values could be specified
+ native import support (in YAML is possible w/ extensions not in the spec https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another https://davidchall.github.io/yaml-includes.html)
- single-line comments
- unclear multi-line string support
- only basic (json) data types (no sets, ordered dicts, ...)
- no type annotation
https://cdcvs.fnal.gov/redmine/attachments/download/29136/quick_start_v3.pdf
https://cdcvs.fnal.gov/redmine/attachments/download/6639/grammar.pdf
https://cdcvs.fnal.gov/redmine/projects/fhicl
This should be compared to YAML
#5 Updated by Marco Mambelli over 1 year ago
Fernando did a Python-YAML comparison with a mocked Factory config and a parser. The results are in github:
https://github.com/fernandolis10/Factory-Configs
Check the testing branch:
https://github.com/fernandolis10/Factory-Configs/blob/testing/config.py
#6 Updated by Marco Mambelli over 1 year ago
- Description updated (diff)
#7 Updated by Marco Mambelli over 1 year ago
- a config module for python:
- http://www.red-dove.com/config-doc/
- https://pypi.org/project/config/
It adds some extensions to JSON:`` execution in dotted notation, e.g. to get environment variables $ variable substitution @ to include an external file
Would be nice to have something similar for YAML
Ansible has some extensions to YAML:
https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html
"{{ variable }}" for variables
YAML spec special chars:
https://yaml.org/spec/1.2/spec.html#id2765878
c-indicator: - ? : list item, key, value , [] {} for flow notation (c-flow-indicator) & * ! anchor, alias, tag # comment | > flow and folded block ' " single and double quoted strings % directive line ` @ reserved for future use cr lf are the only line-break characters sp tab are the only white-space characters
PyYAML allows you to attach custom constructors (such as !include) to the YAML loader
https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another
import yaml import os class Loader(yaml.SafeLoader): def __init__(self, stream): self._root = os.path.split(stream.name)[0] super(Loader, self).__init__(stream) def include(self, node): filename = os.path.join(self._root, self.construct_scalar(node)) with open(filename, 'r') as f: return yaml.load(f, Loader) Loader.add_constructor('!include', Loader.include)
Here a gist with a better python3 implementation and usage example:
https://gist.github.com/joshbode/569627ced3076931b02f
Symphony is a YAML library for PHP that has different extensions:
https://symfony.com/doc/current/components/yaml.html
#8 Updated by Marco Mambelli over 1 year ago
YAML has inheritance:
https://web.archive.org/web/20130213112648/http://blog.101ideas.cz/posts/dry-your-yaml-files.html
https://stackoverflow.com/questions/14184971/more-complex-inheritance-in-yaml
<<
https://medium.com/@kinghuang/docker-compose-anchors-aliases-extensions-a1e4105d70bd
https://medium.com/@sidneyliebrand/the-greatnesses-and-gotchas-of-yaml-5e3377ef0c55
#9 Updated by Marco Mambelli over 1 year ago
- making YAML more robust and add validation via a schema: StrictYAML
- Extended INI format: TOML
- Extended JSON w/ comments JSON5
#10 Updated by Marco Mambelli 9 months ago
- https://github.com/google/jsonnet
- https://jsonnet.org/
- https://tanka.dev/jsonnet/overview
- https://medium.com/@prune998/using-jsonnet-does-not-have-to-be-complex-54b1ad9b21db
- http://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml.html
- https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1.pdf
- https://hub.packtpub.com/introducing-cue-an-open-source-data-constraint-language-that-merges-types-and-values-into-a-single-concept/#:~:text=Inspired%20by%20Google's%20General%20Configuration,involve%20defining%20and%20using%20data.
- https://dhall-lang.org/
- https://nixos.org/nix/manual/#chap-writing-nix-expressions
- skylark or Starlark
- CUE (config and validation)