Feature #23117

Feature #23091: Reliable, flexible and secure logging system for distributed workflows

Additional logging channel

Added by Leonardo Lai over 1 year ago. Updated about 2 months ago.

Work in progress
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)


The idea is to develop a new mechanism enabling glidein scripts (e.g. glidein_startup) to log text/files. Such logs are eventually sent to a remote server.

The tasks includes defining a proper format for the logs (likely JSON), and functions to manipulate them. The log must also contain info about the glidein properties (factory, entries, ...).
Also, the system should work seamlessly with any glidein type (multiglidein/multirestart too).


Feature #23265: Add security mechanisms to the glidein logging channelWork in progressLeonardo Lai

Feature #24140: Implement and deploy a first version of the secure logging channelClosedMarco Mambelli


#1 Updated by Marco Mambelli over 1 year ago

  • Target version changed from v3_4_6 to v3_4_x

#2 Updated by Leonardo Lai over 1 year ago

  • Status changed from New to Work in progress

A function called 'log_write' has been added to, which adds a new entry to the glidein log file in JSON format.
You can record either a string message or the content of a file.

log_write <invoker> <type> <message/filename> <severity>

invoker          -> name of the script calling the function
type             -> text/file
message/filename -> depending on the type, the string or the name of the file
severity         -> debug/info/warn/error

A JSON log entry (aka shard) looks like this:

 "content":"This is an example of warning" 

Metadata of the glidein is automatically prepended on top of the log, in a separate shard.
When the log is empty, the file contains only the metadata, without any event.

The function 'log_coalesce_shards' collects all the existing shards and appends them (in chronological order) to the pre-existing glidein log.
The function 'send_logs_to_remote' forwards the log to a remote server, whose address is currently hardcoded in but will become a configurable parameter soon.

Logging functions can be invoked by any other script after sourcing it (attribute LOGGING_UTILS_SOURCE in glidein_config).


Where is the code?

The functions are in web_base/creation/, branch v34/leo (as of 08/21/19).
Unfortunately most of the code is wrapped in heredoc, so syntax highlighting won't make it more readable.

EDIT (09/05/19): all the functions are now in a standalone file, named logging_utils.source.

Why should I hardcode the invoker name?

There doesn't seem to exist any trivial solution to avoid this.
$0 wouldn't work for sourced files (it returns the name of the sourcer)
$BASH_SOURCE would instead return 'condor_exec.exe' in, which is the alias while it effectively runs on worker nodes; those names would make logs a bit more obscure, imo.

Can I log_write files in a custom directory?

Yes, the filename is actually the relative path w.r.t the Glidein working directory

Should I ever call log_coalesce_shards?

No need, unless you want to merge the shards without sending the log.
Indeed, send_logs_to_remote automatically calls log_coalesce_shards.

Can you provide a minimal example of usage?

logging_utils_source="`grep '^LOGGING_UTILS_SOURCE ' "$glidein_config" | cut -d ' ' -f 2-`" 
source "$logging_utils_source" 
log_setup "$glidein_config" 
log_write "" "text" "This is an example of warning" "warn" 

What about security?

Good point. Authentication between the nodes and the remote server will be addressed soon.

#3 Updated by Leonardo Lai over 1 year ago

  • Start date changed from 08/14/2019 to 09/12/2019
  • Due date set to 09/12/2019

due to changes in a related task: #23265

#4 Updated by Marco Mambelli over 1 year ago

  • Target version changed from v3_4_x to v3_7_x

#5 Updated by Marco Mambelli about 1 year ago

  • Start date changed from 09/12/2019 to 03/06/2020
  • Due date set to 03/06/2020

due to changes in a related task: #23265

Also available in: Atom PDF