Project

General

Profile

Feature #20766

Change configuration handling and configuration files format

Added by Marco Mambelli 12 months ago. Updated 12 days ago.

Status:
New
Priority:
Normal
Assignee:
Fernando Lisboa
Category:
Configuration Management
Target version:
v3_7_x
Start date:
09/06/2018
Due date:
% Done:

0%

Estimated time:
Stakeholders:
Duration:

Description

Factory and Frontend configuration files are in XML
Furthermore they are being modified sometimes during the reconfig/upgrade commands

The new configuration files
- should be easy to write/read
- should allow comments
- should allow inclusion of multiple files
- should never be modified by reload(reconfig) or upgrade or restart

There should be a tool for checking/completing the configuration (different form the reloading). This could modify the file in place or write a different version.

Variables lists like condor_vars.lst should always be used to check correct type/options of the attributes and to set default values (see [#20749])

Some considerations based on current configuration handling in GWMS:
- should be in a different format from XML, easier to write and check (maybe Python, or JSON derivative like YAML)
https://www.zionandzion.com/json-vs-xml-vs-toml-vs-cson-vs-yaml/
- Upgrade could become an option of restart (or always be done at restart)
- Reloading should always happen on restart. And remain an option during run-time

Generic considerations:
- operators should evaluate the formats (e.g. presenting mock-ups in different formats) and pick one
- better if it allows defaults specification (e.g. for all attributes the dafault is... following attributes lists will include only elements that change)
- better if it allows type annotation and verification (e.g. enumerates)

History

#1 Updated by Marco Mambelli 12 months ago

Added consideration:
- the configuration should allow the execution of code to generate some content (dynamic content/plugins)
- the configuration framework should be the same for Factory and Frontend, keeping as much as possible common (parsing, conversion, default managements, plugins, framework) and clearly isolate parts that are different (specific constraint and content)

Formats to consider:
Python, YAML, HJSON, JSON5?

#2 Updated by Marco Mambelli 27 days ago

  • Assignee set to Fernando Lisboa

#4 Updated by Marco Mambelli 12 days ago

Another format that could be considered is FHiCL. It has some interesting extensions in variables handling and allows imports:
- developed and used only at Fermilab (limited use, limited language support, anyway C++ and Python are supported)
+ implicit references: all elements can be referenced using dot notation and seq index) (note the sequential evaluation of references, different from HTCondor)
- everything can be a variable (this can be confusing in long files)
+ binding protection (ignore reassignment or throw error, a way to have constants),
+ prolog sections where default values could be specified
+ native import support (in YAML is possible w/ extensions not in the spec https://stackoverflow.com/questions/528281/how-can-i-include-a-yaml-file-inside-another https://davidchall.github.io/yaml-includes.html)
- single-line comments
- unclear multi-line string support
- only basic (json) data types (no sets, ordered dicts, ...)
- no type annotation

https://cdcvs.fnal.gov/redmine/attachments/download/29136/quick_start_v3.pdf
https://cdcvs.fnal.gov/redmine/attachments/download/6639/grammar.pdf
https://cdcvs.fnal.gov/redmine/projects/fhicl

This should be compared to YAML

#5 Updated by Marco Mambelli 12 days ago

Fernando did a Python-YAML comparison with a mocked Factory config and a parser. The results are in github:
https://github.com/fernandolis10/Factory-Configs

Check the testing branch:
https://github.com/fernandolis10/Factory-Configs/blob/testing/config.py

#6 Updated by Marco Mambelli 12 days ago

  • Description updated (diff)

Also available in: Atom PDF