Project

General

Profile

Feature #23265

Feature #23091: Reliable, flexible and secure logging system for distributed workflows

Feature #23117: Additional logging channel

Add security mechanisms to the glidein logging channel

Added by Leonardo Lai 2 months ago. Updated 19 days ago.

Status:
Work in progress
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
09/12/2019
Due date:
% Done:

80%

Estimated time:
Stakeholders:
Duration:

Description

With the new glidein logging system, logs are occasionally sent from the glideins to a remote http server for storage. Being internet inherently insecure, additional security mechanisms must be adopted to prevent these messages from being intercepted or tampered, and to authenticate all the involved entities.

The implemented system should be relatively lightweight not to compromise the scalability (number of glideins); extremely secure measures are not really required in this case, as the log messages are not meant to contain strictly confidential information. Nevertheless, we don't want anybody in the network to easily read/steal these messages, or forge spurious ones.

The initial idea is to add an encryption layer (SSL/TSL) to protect the traffic, and a JSON Web Tokens for authentication.

auth_http_logserver.py (6.21 KB) auth_http_logserver.py simple python logserver Leonardo Lai, 09/24/2019 11:48 AM

History

#1 Updated by Leonardo Lai 2 months ago

Glideins now exchange messages with the remote server through HTTPS.
The server uses its x509 certificate signed by IGTF CA.

#2 Updated by Leonardo Lai 2 months ago

  • Status changed from New to Work in progress

#3 Updated by Leonardo Lai about 2 months ago

The Factory is now able to issue JSON Web Tokens (JWT) and give them to the glideins, which use them to authenticate with the log server.

A token exists for any valid pair entry-logserver. The tokens are currently generated by the Factory when it starts, and remain valid for approximately 1 week.
Although not yet implemented, the system can be extended to allow Frontends generating the tokens and pass them to the glideins through the Factory. This would be especially useful when the log server runs on the same machine of the Frontend, so that the secret key (for token issue and validation) is local.

The log servers can be configured in glideinWMS.xml for each entry:

<attr const="False" glidein_publish="False" job_publish="False" name="LOG_RECIPIENTS_FACTORY" parameter="True" publish="True" type="string" value="https://fermicloud152.fnal.gov/log https://fermicloud365.fnal.gov" />

The tokens are securely transferred to the glideins with Condor encrypt_input_files ; since the list of files to submit is static (cgWCreate.py), the names of the files to send must be known a priori, when reconfiguring the Factory. However, because a the Frontend could change log recipients dynamically (i.e. while Factory is running), we must send a fixed number of files with fixed names.
Specifically, two files are sent: tokens.tgz, which is an archive of tokens, and url_dirs.desc which is a utility dictionary to traslate server URLs to the respective directory names.

When a glidein sends a log to the server (HTTP PUT request), it includes the JWT in its header.

Token renewal will be implemented soon, a cronjob is probably the best choice.

The experimental code is available in branch v34/leo_token.
Any kind of HTTPS server can sit on the receiving side: I attach a simple Python one that can be easily used for testing.

#4 Updated by Leonardo Lai about 2 months ago

A recap for whoever will takeover this work after my internship ends.
My code is in branch v34/leo_token.

Things left to implement:

1) token renewal: maybe a cronjob in the factory? To make sure that a glidein gets a token that will be valid for all the glidein lifecycle (from spawn to death), you need to comply with the formula:

T_token > T_renewal + T_glidein

where

T_token ----> token validity at issuance
T_renewal --> renewal period
T_glidein --> maximum expected lifespan for a glidein

2) Automatic secret key exchange between issuer (factory) and verifier (logserver). Right now you need to manually put the keys in those hosts. It would be even easier to switch to a private-public key system (just change the protocol field in JWT encode/decode functions); however, the pyjwt library requires the 'cryptography' module to work, which is available only from Python 2.7.
3) Add support for token generation by the frontend: aside from token generation, which is the easy part (just copy the factory code), you need to first send those tokens to the factory updating the tokens.tgz and url_dirs.desc files accordingly. Server URLs should be configurable with an attribute.

Useful resources
https://jwt.io/ ---> manually create and verify tokens
https://tools.ietf.org/html/rfc7519 ---> RFC7519, with full details about JWT

#5 Updated by Marco Mambelli about 1 month ago

  • Target version changed from v3_4_7 to v3_6_1

#6 Updated by Marco Mambelli 19 days ago

  • Target version changed from v3_6_1 to v3_7


Also available in: Atom PDF