Project

General

Profile

Feature #23117

Feature #23091: Reliable, flexible and secure logging system for distributed workflows

Additional logging channel

Added by Leonardo Lai about 1 month ago. Updated 6 days ago.

Status:
Work in progress
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
09/12/2019
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Stakeholders:
Duration:

Description

The idea is to develop a new mechanism enabling glidein scripts (e.g. glidein_startup) to log text/files. Such logs are eventually sent to a remote server.

The tasks includes defining a proper format for the logs (likely JSON), and functions to manipulate them. The log must also contain info about the glidein properties (factory, entries, ...).
Also, the system should work seamlessly with any glidein type (multiglidein/multirestart too).


Subtasks

Feature #23265: Add security mechanisms to the glidein logging channelWork in progressLeonardo Lai

History

#1 Updated by Marco Mambelli about 1 month ago

  • Target version changed from v3_4_6 to v3_4_x

#2 Updated by Leonardo Lai 28 days ago

  • Status changed from New to Work in progress

A function called 'log_write' has been added to glidein_startup.sh, which adds a new entry to the glidein log file in JSON format.
You can record either a string message or the content of a file.

log_write <invoker> <type> <message/filename> <severity>

invoker          -> name of the script calling the function
type             -> text/file
message/filename -> depending on the type, the string or the name of the file
severity         -> debug/info/warn/error

A JSON log entry (aka shard) looks like this:

{"invoker":"glidein_memory_setup.sh", 
 "pid":"11269", 
 "timestamp":"2019-08-19T17:00:56-05:00", 
 "severity":"warn", 
 "type":"text", 
 "filename":"", 
 "content":"This is an example of warning" 
}

Metadata of the glidein is automatically prepended on top of the log, in a separate shard.
When the log is empty, the file contains only the metadata, without any event.

The function 'log_coalesce_shards' collects all the existing shards and appends them (in chronological order) to the pre-existing glidein log.
The function 'send_logs_to_remote' forwards the log to a remote server, whose address is currently hardcoded in glidein_startup.sh but will become a configurable parameter soon.

Logging functions can be invoked by any other script after sourcing it (attribute LOGGING_UTILS_SOURCE in glidein_config).


FAQ

Where is the code?

The functions are in web_base/creation/glidein_startup.sh, branch v34/leo (as of 08/21/19).
Unfortunately most of the code is wrapped in heredoc, so syntax highlighting won't make it more readable.

EDIT (09/05/19): all the functions are now in a standalone file, named logging_utils.source.

Why should I hardcode the invoker name?

There doesn't seem to exist any trivial solution to avoid this.
$0 wouldn't work for sourced files (it returns the name of the sourcer)
$BASH_SOURCE would instead return 'condor_exec.exe' in glidein_startup.sh, which is the alias while it effectively runs on worker nodes; those names would make logs a bit more obscure, imo.

Can I log_write files in a custom directory?

Yes, the filename is actually the relative path w.r.t the Glidein working directory

Should I ever call log_coalesce_shards?

No need, unless you want to merge the shards without sending the log.
Indeed, send_logs_to_remote automatically calls log_coalesce_shards.

Can you provide a minimal example of usage?

logging_utils_source="`grep '^LOGGING_UTILS_SOURCE ' "$glidein_config" | cut -d ' ' -f 2-`" 
source "$logging_utils_source" 
log_setup "$glidein_config" 
log_write "foobar.sh" "text" "This is an example of warning" "warn" 
send_logs_to_remote

What about security?

Good point. Authentication between the nodes and the remote server will be addressed soon.

#3 Updated by Leonardo Lai 6 days ago

  • Start date changed from 08/14/2019 to 09/12/2019
  • Due date set to 09/12/2019

due to changes in a related task: #23265



Also available in: Atom PDF