With the new glidein logging system, logs are occasionally sent from the glideins to a remote http server for storage. Being internet inherently insecure, additional security mechanisms must be adopted to prevent these messages from being intercepted or tampered, and to authenticate all the involved entities.
The implemented system should be relatively lightweight not to compromise the scalability (number of glideins); extremely secure measures are not really required in this case, as the log messages are not meant to contain strictly confidential information. Nevertheless, we don't want anybody in the network to easily read/steal these messages, or forge spurious ones.
The initial idea is to add an encryption layer (SSL/TSL) to protect the traffic, and a JSON Web Tokens for authentication.
#3 Updated by Leonardo Lai 4 months ago
The Factory is now able to issue JSON Web Tokens (JWT) and give them to the glideins, which use them to authenticate with the log server.
A token exists for any valid pair entry-logserver. The tokens are currently generated by the Factory when it starts, and remain valid for approximately 1 week.
Although not yet implemented, the system can be extended to allow Frontends generating the tokens and pass them to the glideins through the Factory. This would be especially useful when the log server runs on the same machine of the Frontend, so that the secret key (for token issue and validation) is local.
The log servers can be configured in glideinWMS.xml for each entry:
<attr const="False" glidein_publish="False" job_publish="False" name="LOG_RECIPIENTS_FACTORY" parameter="True" publish="True" type="string" value="https://fermicloud152.fnal.gov/log https://fermicloud365.fnal.gov" />
The tokens are securely transferred to the glideins with Condor
encrypt_input_files ; since the list of files to submit is static (cgWCreate.py), the names of the files to send must be known a priori, when reconfiguring the Factory. However, because a the Frontend could change log recipients dynamically (i.e. while Factory is running), we must send a fixed number of files with fixed names.
Specifically, two files are sent:
tokens.tgz, which is an archive of tokens, and
url_dirs.desc which is a utility dictionary to traslate server URLs to the respective directory names.
When a glidein sends a log to the server (HTTP PUT request), it includes the JWT in its header.
Token renewal will be implemented soon, a cronjob is probably the best choice.
The experimental code is available in branch v34/leo_token.
Any kind of HTTPS server can sit on the receiving side: I attach a simple Python one that can be easily used for testing.
#4 Updated by Leonardo Lai 4 months ago
A recap for whoever will takeover this work after my internship ends.
My code is in branch
1) token renewal: maybe a cronjob in the factory? To make sure that a glidein gets a token that will be valid for all the glidein lifecycle (from spawn to death), you need to comply with the formula:
T_token > T_renewal + T_glidein where T_token ----> token validity at issuance T_renewal --> renewal period T_glidein --> maximum expected lifespan for a glidein
2) Automatic secret key exchange between issuer (factory) and verifier (logserver). Right now you need to manually put the keys in those hosts. It would be even easier to switch to a private-public key system (just change the protocol field in JWT encode/decode functions); however, the pyjwt library requires the 'cryptography' module to work, which is available only from Python 2.7.
3) Add support for token generation by the frontend: aside from token generation, which is the easy part (just copy the factory code), you need to first send those tokens to the factory updating the
url_dirs.descfiles accordingly. Server URLs should be configurable with an attribute.