GlideinWMS

Added consideration:
- the configuration should allow the execution of code to generate some content (dynamic content/plugins)
- the configuration framework should be the same for Factory and Frontend, keeping as much as possible common (parsing, conversion, default managements, plugins, framework) and clearly isolate parts that are different (specific constraint and content)

Formats to consider:
Python, YAML, HJSON, JSON5?

Here some links I found about configuration files formats:
https://stackoverflow.com/questions/1773805/how-can-i-parse-a-yaml-file-in-python
https://en.wikipedia.org/wiki/Comparison_of_data-serialization_formats
https://martin-thoma.com/configuration-files-in-python/
https://stackoverflow.com/questions/791761/xml-for-configuration-files-why
https://yaml.org/spec/1.2/spec.html#id2759732

Including an interesting library:
https://github.com/MartinThoma/cfg_load

Some things to keep in mind about YAML:
- spec v1.2 comes in 3 flavors (Failsafe, JSON and Core schema) and is more strict of 1.1 (especially the JSON schema) that is a superset of JSON
- use true, false, null (empty scalar may be an empty string)
https://perlpunk.github.io/slides.tpcig2018/yamlpp/slide021.html
http://blogs.perl.org/users/tinita/2018/01/introduction-to-yaml-schemas-and-tags.html
https://yaml.readthedocs.io/en/latest/pyyaml.html

Another format that could be considered is FHiCL. It has some interesting extensions in variables handling and allows imports:
- developed and used only at Fermilab (limited use, limited language support, anyway C++ and Python are supported)
+ implicit references: all elements can be referenced using dot notation and seq index) (note the sequential evaluation of references, different from HTCondor)
- everything can be a variable (this can be confusing in long files)
+ binding protection (ignore reassignment or throw error, a way to have constants),
+ prolog sections where default values could be specified
+ native import support (in YAML is possible w/ extensions not in the spec https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another https://davidchall.github.io/yaml-includes.html)
- single-line comments
- unclear multi-line string support
- only basic (json) data types (no sets, ordered dicts, ...)
- no type annotation

https://cdcvs.fnal.gov/redmine/attachments/download/29136/quick_start_v3.pdf
https://cdcvs.fnal.gov/redmine/attachments/download/6639/grammar.pdf
https://cdcvs.fnal.gov/redmine/projects/fhicl

This should be compared to YAML

Fernando did a Python-YAML comparison with a mocked Factory config and a parser. The results are in github:
https://github.com/fernandolis10/Factory-Configs

Check the testing branch:
https://github.com/fernandolis10/Factory-Configs/blob/testing/config.py

Adding some new information:

• a config module for python:
  • http://www.red-dove.com/config-doc/
  • https://pypi.org/project/config/
    It adds some extensions to JSON:
    

    `` execution in dotted notation, e.g. to get environment variables
    $ variable substitution
    @ to include an external file
    

Would be nice to have something similar for YAML

Ansible has some extensions to YAML:
https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html

"{{ variable }}" for variables

YAML spec special chars:
https://yaml.org/spec/1.2/spec.html#id2765878

c-indicator:
- ? :  list item, key, value
, [] {}  for flow notation (c-flow-indicator)
& * !  anchor, alias, tag
# comment
| > flow and folded block
' "  single and double quoted strings
% directive line
` @ reserved for future use
cr lf are the only line-break characters
sp tab are the only white-space characters

PyYAML allows you to attach custom constructors (such as !include) to the YAML loader
https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another

import yaml
import os

class Loader(yaml.SafeLoader):

    def __init__(self, stream):

        self._root = os.path.split(stream.name)[0]

        super(Loader, self).__init__(stream)

    def include(self, node):

        filename = os.path.join(self._root, self.construct_scalar(node))

        with open(filename, 'r') as f:
            return yaml.load(f, Loader)

Loader.add_constructor('!include', Loader.include)

Here a gist with a better python3 implementation and usage example:
https://gist.github.com/joshbode/569627ced3076931b02f

Symphony is a YAML library for PHP that has different extensions:
https://symfony.com/doc/current/components/yaml.html

Hiera - yaml hierarchy from puppet

• https://puppet.com/docs/puppet/latest/hiera_config_yaml_5.html
• https://forge.puppet.com/puppet/hiera
  Python modules for hiera:
• https://github.com/adobe/himl
• https://pypi.org/project/piera/
• https://pypi.org/project/hiera-py/

YAML has inheritance:
https://web.archive.org/web/20130213112648/http://blog.101ideas.cz/posts/dry-your-yaml-files.html
https://stackoverflow.com/questions/14184971/more-complex-inheritance-in-yaml

<<

https://medium.com/@kinghuang/docker-compose-anchors-aliases-extensions-a1e4105d70bd
https://medium.com/@sidneyliebrand/the-greatnesses-and-gotchas-of-yaml-5e3377ef0c55

Some more considerations:

• making YAML more robust and add validation via a schema: StrictYAML
  • https://hitchdev.com/strictyaml/features-removed/
  • https://github.com/crdoconnor/strictyaml
  • https://hitchdev.com/strictyaml/why/node-anchors-and-references-removed/
• Extended INI format: TOML
  • https://github.com/toml-lang/toml
• Extended JSON w/ comments JSON5
  • https://json5.org/
  • https://www.reddit.com/r/Python/comments/8gqo70/pyjson5_a_json5_serializer_and_parser_library_for/
  • https://pypi.org/project/json5/

Discussions:

• https://news.ycombinator.com/item?id=17358103
• https://www.arp242.net/yaml-config.html

Decision Engine decided to go w/ jsonnet

• https://github.com/google/jsonnet
• https://jsonnet.org/
• https://tanka.dev/jsonnet/overview
• https://medium.com/@prune998/using-jsonnet-does-not-have-to-be-complex-54b1ad9b21db
• http://leebriggs.co.uk/blog/2019/02/07/why-are-we-templating-yaml.html

Config languages in general, GCL

• https://pure.tue.nl/ws/portalfiles/portal/46927079/638953-1.pdf
• https://hub.packtpub.com/introducing-cue-an-open-source-data-constraint-language-that-merges-types-and-values-into-a-single-concept/#:~:text=Inspired%20by%20Google's%20General%20Configuration,involve%20defining%20and%20using%20data.

Others, discarded:

• https://dhall-lang.org/
• https://nixos.org/nix/manual/#chap-writing-nix-expressions
• skylark or Starlark
  • https://github.com/google/starlark-go
  • https://github.com/google/starlark-go/blob/master/doc/spec.md
  • https://docs.bazel.build/versions/master/skylark/language.html
• CUE (config and validation)
  • https://github.com/cuelang/cue/blob/master/doc/ref/spec.md
  • https://cue.googlesource.com/cue/+/HEAD/doc/tutorial/kubernetes/README.md
  • https://cue.googlesource.com/cue