Ingesting new modules

Modules are organized in folders, each folder containing subfolders for different module versions. One might or example want to run a dummy version instead, or compare different algorithms side-by-side. In that case, one can specify in glue.ini (or glue.mine.ini) in the corresponding section which version is to be used.

Settle on input, output and parameters


All the input a module needs it can access from the data bank at the beginning of the program using these commands:

import sys
import archive

data_bank_location = '../../../data/data_bank.h5'
with archive.archive(data_bank_location,'r') as ar:
    local_variable = ar['/folder/input_variable_name']

Please refer to the existing module versions' code to see what data can be accessed. The documentation in the wiki needs updating.


Similarly, at the end of the module run, its output is written to the data bank like this:

with archive.archive(data_bank_location,'a') as ar:
    ar['/folder2/variable_name2'] = local_variable2

Note the option 'a' for 'append'. Do not use 'w' (as in 'write'): it will delete all other contents of the data bank!


Parameters are defined in a .ini file in the module folder. Those are quantities to fiddle with, defining how the data is to be handled. The parameter file's contents are written automatically to the data bank at the beginning of the pipeline run, so they can be accessed by the module. E. g. if your ini file looks like this:

variable = 42

and you specify in glue.ini (or glue.mine.ini)

folder = Module_Name/Module_Name_Author
command = python
parameter = module_parametes.ini

then you can import the parameters inside using

with archive.archive(data_bank_location,'r') as ar:
    parameter1 = ar['/Module_Name/parameter1']
    parameter2 = ar['/Module_Name/parameter2']

Parameters can be int, floats, bools, strings, lists and dictionaries. Just write them in the parameter file as you would in Python. You can use # for comments.