Feature #23265

Feature #23091: Reliable, flexible and secure logging system for distributed workflows

Feature #23117: Additional logging channel

Add security mechanisms to the glidein logging channel

Added by Leonardo Lai over 1 year ago. Updated about 2 months ago.

Work in progress
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)


With the new glidein logging system, logs are occasionally sent from the glideins to a remote http server for storage. Being internet inherently insecure, additional security mechanisms must be adopted to prevent these messages from being intercepted or tampered, and to authenticate all the involved entities.

The implemented system should be relatively lightweight not to compromise the scalability (number of glideins); extremely secure measures are not really required in this case, as the log messages are not meant to contain strictly confidential information. Nevertheless, we don't want anybody in the network to easily read/steal these messages, or forge spurious ones.

The initial idea is to add an encryption layer (SSL/TSL) to protect the traffic, and a JSON Web Tokens for authentication. (6.21 KB) simple python logserver Leonardo Lai, 09/24/2019 11:48 AM


Feature #24140: Implement and deploy a first version of the secure logging channelClosedMarco Mambelli


#1 Updated by Leonardo Lai over 1 year ago

Glideins now exchange messages with the remote server through HTTPS.
The server uses its x509 certificate signed by IGTF CA.

#2 Updated by Leonardo Lai over 1 year ago

  • Status changed from New to Work in progress

#3 Updated by Leonardo Lai over 1 year ago

The Factory is now able to issue JSON Web Tokens (JWT) and give them to the glideins, which use them to authenticate with the log server.

A token exists for any valid pair entry-logserver. The tokens are currently generated by the Factory when it starts, and remain valid for approximately 1 week.
Although not yet implemented, the system can be extended to allow Frontends generating the tokens and pass them to the glideins through the Factory. This would be especially useful when the log server runs on the same machine of the Frontend, so that the secret key (for token issue and validation) is local.

The log servers can be configured in glideinWMS.xml for each entry:

<attr const="False" glidein_publish="False" job_publish="False" name="LOG_RECIPIENTS_FACTORY" parameter="True" publish="True" type="string" value="" />

The tokens are securely transferred to the glideins with Condor encrypt_input_files ; since the list of files to submit is static (, the names of the files to send must be known a priori, when reconfiguring the Factory. However, because a the Frontend could change log recipients dynamically (i.e. while Factory is running), we must send a fixed number of files with fixed names.
Specifically, two files are sent: tokens.tgz, which is an archive of tokens, and url_dirs.desc which is a utility dictionary to traslate server URLs to the respective directory names.

When a glidein sends a log to the server (HTTP PUT request), it includes the JWT in its header.

Token renewal will be implemented soon, a cronjob is probably the best choice.

The experimental code is available in branch v34/leo_token.
Any kind of HTTPS server can sit on the receiving side: I attach a simple Python one that can be easily used for testing.

#4 Updated by Leonardo Lai over 1 year ago

A recap for whoever will takeover this work after my internship ends.
My code is in branch v34/leo_token.

Things left to implement:

1) token renewal: maybe a cronjob in the factory? To make sure that a glidein gets a token that will be valid for all the glidein lifecycle (from spawn to death), you need to comply with the formula:

T_token > T_renewal + T_glidein


T_token ----> token validity at issuance
T_renewal --> renewal period
T_glidein --> maximum expected lifespan for a glidein

2) Automatic secret key exchange between issuer (factory) and verifier (logserver). Right now you need to manually put the keys in those hosts. It would be even easier to switch to a private-public key system (just change the protocol field in JWT encode/decode functions); however, the pyjwt library requires the 'cryptography' module to work, which is available only from Python 2.7.
3) Add support for token generation by the frontend: aside from token generation, which is the easy part (just copy the factory code), you need to first send those tokens to the factory updating the tokens.tgz and url_dirs.desc files accordingly. Server URLs should be configurable with an attribute.

Useful resources ---> manually create and verify tokens ---> RFC7519, with full details about JWT

#5 Updated by Marco Mambelli over 1 year ago

  • Target version changed from v3_4_7 to v3_6_1

#6 Updated by Marco Mambelli about 1 year ago

  • Target version changed from v3_6_1 to v3_7

#7 Updated by Marco Mambelli 11 months ago

  • Start date changed from 09/12/2019 to 03/06/2020
  • Due date set to 03/06/2020

due to changes in a related task: #24140

#8 Updated by Marco Mambelli 11 months ago

  • Target version changed from v3_7 to v3_7_1

#9 Updated by Dennis Box 2 months ago

  • Target version changed from v3_7_1 to v3_7_2

#10 Updated by Marco Mambelli about 2 months ago

  • Target version changed from v3_7_2 to v3_7_3

Also available in: Atom PDF