Project

General

Profile

Feature #23092

Use token-auth for Glideins authentication and add support for sci-token

Added by Marco Mambelli 3 months ago. Updated 23 days ago.

Status:
New
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
09/16/2019
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Stakeholders:

OSG

Duration:

Description

OSG plans to drop support for x509 by January 2020
GlideinWMS should start using HTCondor's token-auth for its internal authentication (the Frontend requests a token to the Collector and forwards it to the Glidein via Factory)
Support for sci-token should be added to GlideinWMS
This development should happen in the 3.5 series

Marco had a meeting w/ Brian Bockelman and Brian Lin. Attached are some notes.
Here the action Items that will drive this ticket (GWMS refers to a GlideinWMS developer - the owner of this ticket):
  • OSG will make available an RPM of HTCondor 8.9.2 with SciTokens ASAP.
  • In the meantime, GWMS will take 8.9.2 from htcondor.org without SciTokens.
  • GWMS can immediately start working on TOKEN auth completely internal to gWMS.
  • We think this is doable by early September.
  • GWMS and Derek Weitzel will work to help GWMS generate a public/private keypair and post the public portion to the https://urldefense.proofpoint.com/v2/url?u=https-3A__scitokens.org_dteam&d=DwIFAg&c=gRgGjJ3BkIsb5y6s49QqsA&r=EF06-Wh4L9CNLgD8bnIjNQ&m=whXJbta2PxtV6ms20Mx4w0YjGQU_rJFTNATEoFx-8j0&s=HCxMp32H-c88lhJPfcWUWcdBNzPOLfrl91tgRFKSOHg&e= issuer.
    • Also Derek will help with the CLI to generate tokens with the goal of having gWMS use the scitokens python API directly eventually.
    • Once this is done, GWMS can generate SciTokens but has no CE to submit them to. Everything can be tested but the final "submit pilot to CE" step.
  • OSG will aim to provide a CE that accepts the dteam SciTokens near the end of August.
    • Goal is to have a CE package in osg-development by early September.
  • At least GWMS, Brian, and Brian will aim to have a "hackathon" day at FNAL in early October for this project to either finish late deliverables or put together a demo.

Here the link for the HTCondor development repo (for 8.9.2): https://research.cs.wisc.edu/htcondor/instructions/el/7/development/

tokenauth-notes1.pdf (345 KB) tokenauth-notes1.pdf Marco Mambelli, 08/09/2019 02:04 AM
tokenauth-notes2.pdf (691 KB) tokenauth-notes2.pdf Marco Mambelli, 08/09/2019 02:04 AM
frontend.23092.xml (9.9 KB) frontend.23092.xml Dennis Box, 10/22/2019 07:02 AM
glideinWMS.23092.xml (4.47 KB) glideinWMS.23092.xml Dennis Box, 10/22/2019 07:02 AM
token_entries.23092.xml (5.03 KB) token_entries.23092.xml Dennis Box, 10/22/2019 07:03 AM

Subtasks

Support #23278: Condor 8.9.2 configuration changesNewDennis Box

History

#1 Updated by Dennis Box 3 months ago

I realized this ticket is where some of my work notes should go so they get preserved.

To upgrade a working factory/frontend to condor 8.9.2

  1. yum install --enablerepo osg-upcoming-development condor
  2. add this to condor config ( /etc/condor/config.d/03_gwms_local.config)
    For Frontend:
    ALLOW_READ = *
    ALLOW_DAEMON=$(ALLOW_WRITE)
    For Factory:
    ALLOW_READ = *
    ALLOW_DAEMON=$(ALLOW_WRITE)
    ALLOW_ADVERTISE_MASTER = $(ALLOW_DAEMON)
  3. systemctl restart condor; systemctl reload gwms(factory, frontend) <==== should now work with condor 8.9.2 and GSI authentication

Generating a Condor Token

  • Tokens design document https://docs.google.com/document/d/10uSYp6sePbHPFzKmkTMAwr1uKzuSxebr_Om89IUgyUs/edit
  • JWT (Json Web Tokens) RFC https://tools.ietf.org/html/rfc7519
  • To generate a condor token as root:
    • condor_store_cred -f /etc/condor/passwords.d/password
    • set SEC_PASSWORD_FILE=/etc/condor/passwords.d/password; condor_reconfig
    • condor_token_create -identity (example: condor@$HOSTNAME) -lifetime (seconds) -authz (auth_options) > tokenfile
  • My task (as I currently understand it) is now to generate condor tokens with the correct identities and authz options to allow the Frontend collector to authenticate with the startds in the glideins on the CE.

#2 Updated by Dennis Box 3 months ago

Generating SciTokens

  • Getting SciTokens to work with condor/glideinwms is a step that happens after I get Condor Tokens to work, but generating a SciToken is not hard:
  1. yum install python2-scitokens
  2. generate some keys
    scitokens-admin-create-key --create-keys --pem-private > /tmp/test.scitoken.private.pem
    scitokens-admin-create-key --private-keyfile /tmp/test.scitoken.private.pem --jwks-private > /tmp/test.scitoken.private.jwks
    scitokens-admin-create-key --private-keyfile /tmp/test.scitoken.private.pem --jwks-public > /tmp/test.scitoken.public.jwks
  3. copy the keys over to a web server (say for sake of argument jobsub.fnal.gov)
    cd /var/www/html
    mkdir -p oauth2/certs/.well-known
    create file oauth2/certs/.well-known/openid-configuration with this content:

      {
         "issuer":"https://jobsub.fnal.gov",
         "jwks_uri":"https://jobsub.fnal.gov/oauth2/certs" 
      }
      

    copy test.scitoken.public.jwks to /oauth2/certs directory

  4. Generate a token
    scitokens-admin-create-token --key_id 1d92 --keyfile /tmp/test.scitoken.private.pem --issuer https://jobsub.fnal.gov sub=htcondor 'scope=condor:/READ condor:/WRITE condor:/ALLOW' > condor.token

#3 Updated by Dennis Box 3 months ago

Condor Authentication with Tokens

  • TOKEN and SCITOKEN are both valid settings for SEC_DEFAULT_AUTHENTICATION_METHODS starting with 8.9.2
  • SEC_(DAEMON)_AUTHENTICATION_METHODS now exists for all the condor daemons, so we can customize authentication to a fine grained level if desired.
  • condor_token_create -authz input arguments are discussed here : https://htcondor.readthedocs.io/en/v8_9_2/admin-manual/security.html, for example -authz ADVERTISE_STARTD

#4 Updated by Brian Lin 3 months ago

Dennis Box wrote:

Condor Authentication with Tokens

  • TOKEN and SCITOKEN are both valid settings for SEC_DEFAULT_AUTHENTICATION_METHODS starting with 8.9.2

I believe that unrecognized values in SEC_*_AUTHENTICATION_METHODS are just ignored so you shouldn't have to worry about using if statements in the HTCondor config.

#5 Updated by Dennis Box 3 months ago

User Authorization with Tokens for Job Submission

  • thus far, factory and frontend have had SEC_DEFAULT_AUTHENTICATION_METHODS = FS,GSI
  • change this setting to SEC_DEFAULT_AUTHENTICATION_METHODS = TOKEN,FS,GSI . Restart.
    • frontend, factory, and CEs still play together nicely, starting glideins after a user job submission
  • Add setting SEC_CLIENT_AUTHENTICATION_METHODS = TOKEN,GSI to frontend condor configuration
    • we now need a token to submit, as this 'helpful' error message informs us:
      [dbox@fermicloud096 testjobs]$ condor_submit testjob.singularity.jdf 
      Submitting job(s).
      ERROR: Failed to create proc
      
    • I have ALL_DEBUG = D_SECURITY D_COMMAND set, there is probably something in the log that matches this but I cant find it.
      The error is clearly because I need a token. After playing with the condor_token_ commands for a while this is how I finally enabled submission:
      
      + mkdir -p /cloud/login/dbox/.condor/tokens.d
      + touch /cloud/login/dbox/.condor/tokens.d/dbox.token
      + condor_token_create -identity dbox@fermicloud096.fnal.gov >> /cloud/login/dbox/.condor/tokens.d/dbox.token
      + chmod 400 /cloud/login/dbox/.condor/tokens.d/dbox.token
      + chown -R dbox /cloud/login/dbox/.condor
      
      
  • this token is pretty 'wide open' , no expiration date or internal limit to what it is allowed to do. Obviously not suitable for a production environment. I am just trying to get things up and running, and this does allow submissions.
  • to get jobs to start I had to do this as root:
     condor_token_create -identity condor@$HOSTNAME        -authz ADMINISTRATOR > /etc/condor/tokens.d/admin.token
    
  • I think this token being 'wide open' is less of a problem than the user token, its somewhat like having an encrypted root password in /etc/passwd on unix. Not ideal but not horrifically bad like the user token.

Increasing the scope of daemons that use TOKEN instead of FS or GSI

  • In order to understand this new security model and TOKEN authentications place in it, I am tightening down the SEC_*_AUTHENTICATION_METHODS one by one until something breaks, then seeing what I need to do to fix it.
  • Turning these knobs 1 by 1 on the frontend still allows me a fully working gwms submission system with the above 2 (probably overly generous) tokens
    • SEC_CLIENT_AUTHENTICATION_METHODS = TOKEN,GSI
    • etc through all the knobs, eliminating FS authentication.
    • end result is a working frontend that uses GSI and TOKEN authentication only, glideins start, phone home, user jobs run
  • Now start turning off GSI to see how far we can take this without further code modification

#6 Updated by Dennis Box 2 months ago

  • Start date changed from 08/09/2019 to 09/16/2019
  • Due date changed from 08/30/2019 to 09/16/2019

due to changes in a related task: #23278

#7 Updated by Marco Mambelli 2 months ago

  • Target version changed from v3_5_x to v3_7

#8 Updated by Dennis Box 2 months ago

Recipe for Sending Glideins That Connect Back With Token Auth

  • This is not 'production ready', it is more of a proof of concept
  • Requirements:
    • Glidein entry point runs glideins with condor 8.9.2
    • A token, osg.token has been created and placed in /etc/gwms-frontend/creds/osg.token
      • no work has (yet) been done on securing osg.token, its permissions are too broad for anything other than demonstration purposes
      • if osg.token has an expiration date, a mechanism must be in place to renew it periodically
  • add to the <attrs> section of frontend.xml
          <attr name="SEC_DEFAULT_AUTHENTICATION_METHODS" comment="glidein condor_config setting, use TOKEN auth exclusively" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="TOKEN"/>
          <attr name="SEC_TOKEN_DIRECTORY" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(LOCAL_DIR)/usertokens.d"/>
          <attr name="SEC_TOKEN_SYSTEM_DIRECTORY" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(LOCAL_DIR)/systokens.d"/>
          <attr name="SEC_PASSWORD_DIRECTORY" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(LOCAL_DIR)/passwords.d"/>
          <attr name="SEC_PASSWORD_FILE" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(SEC_PASSWORD_DIRECTORY)/password"/>
          <attr name="TOKEN_FILE" comment="goes in SEC_TOKEN_DIRECTORY" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="osg.token"/>
          <attr name="EXTRA_LIB_LIST" comment="libraries needed for token auth by condor 8.9.2" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="libSciTokens.so.0.0.2 libmunge.so.2.0"/>
    
  • add to <files> section of frontend.xml
                <file absfname="/usr/lib64/libSciTokens.so.0.0.2" comment="needed for TOKEN auth in glidein" after_entry="False" const="True" executable="False" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
                </file>
                <file absfname="/usr/lib64/libmunge.so.2.0.0" comment="needed for TOKEN auth in glidein" after_entry="False" const="True" executable="False" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
                </file>
                <file absfname="/etc/gwms-frontend/creds/osg.token" comment="actual token that is used for authorization" after_entry="False" const="True" executable="False" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
                </file>
                <file absfname="/var/lib/gwms-frontend/web-base/frontend/token_setup.sh" comment="script to create dirs and move uploaded files correct places in glidein" after_entry="False" const="True" executable="True" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
                </file>
    
    
  • contents of token_setup.sh, that creates needed directories and makes sure uploaded files are in correct place
#!/bin/bash 
# 

glidein_config_val () {
   grep "^$1 " $glidein_config | cut -d ' ' -f 2-
}

if [ "$glidein_config" = "" ]; then
        glidein_config="$1" 
fi

GWMS_THIS_SCRIPT="`basename "$0"`" 
GWMS_THIS_SCRIPT_DIR="`dirname "$0"`" 

GLIDEIN_WORK_DIR=$(glidein_config_val GLIDEIN_WORK_DIR)
LOCAL_DIR=$(dirname $GLIDEIN_WORK_DIR)
EXTRA_LIB_LIST=$(glidein_config_val EXTRA_LIB_LIST)
TOKEN_FILE=$(glidein_config_val TOKEN_FILE)
CONDOR_DIR=$(glidein_config_val CONDOR_DIR)

SEC_TOKEN_DIRECTORY=$(echo $(glidein_config_val SEC_TOKEN_DIRECTORY) | sed -e 's/.*\///')
SEC_TOKEN_DIRECTORY="${LOCAL_DIR}/${SEC_TOKEN_DIRECTORY}" 

SEC_TOKEN_SYSTEM_DIRECTORY=$(echo $(glidein_config_val SEC_TOKEN_SYSTEM_DIRECTORY)| sed -e 's/.*\///')
SEC_TOKEN_SYSTEM_DIRECTORY="${LOCAL_DIR}/${SEC_TOKEN_SYSTEM_DIRECTORY}" 

SEC_PASSWORD_DIRECTORY=$(echo $(glidein_config_val SEC_PASSWORD_DIRECTORY) | sed -e 's/.*\///')
SEC_PASSWORD_DIRECTORY="${LOCAL_DIR}/${SEC_PASSWORD_DIRECTORY}" 

SEC_PASSWORD_FILE=$(echo $(glidein_config_val SEC_PASSWORD_FILE) | sed -e 's/.*\///')
SEC_PASSWORD_FILE="${SEC_PASSWORD_DIRECTORY}/${SEC_PASSWORD_FILE}" 

mkdir -p $SEC_TOKEN_SYSTEM_DIRECTORY
mkdir -p $SEC_TOKEN_DIRECTORY
mkdir -p $SEC_PASSWORD_DIRECTORY
touch $SEC_PASSWORD_FILE
cp $GWMS_THIS_SCRIPT_DIR/$TOKEN_FILE $SEC_TOKEN_DIRECTORY
mv $GWMS_THIS_SCRIPT_DIR/$TOKEN_FILE $SEC_TOKEN_SYSTEM_DIRECTORY
for LIB in $EXTRA_LIB_LIST; do
    mv $GWMS_THIS_SCRIPT_DIR/$LIB ${CONDOR_DIR}/lib
done
cd ${CONDOR_DIR}/lib
ln -s libSciTokens.so.0.0.2 libSciTokens.so.0
ln -s libSciTokens.so.0.0.2 libSciTokens.so
ln -s libmunge.so.2.0.0 libmunge.so.2
cd -

  • systemctl reload gwms-frontend and now glideins auth back to the frontend collector using TOKEN auth

#9 Updated by Dennis Box about 1 month ago

Improved Setup

Current code changes are on git branch v37/23092. The previous configuration was passing tokens in cleartext, they are now encrypted and handled with the same mechanism that proxies and cert pairs use.

The frontend.xml credentials section is configured with a token to communicate with the CE alongside the proxy that is used to talk to the factory:

      <credentials>
         <credential absfname="/etc/gwms-frontend/creds/vo_proxy" security_class="frontend" trust_domain="condor_token" type="grid_proxy"/>
         <credential absfname="/etc/gwms-frontend/creds/osg.token" security_class="frontend_token" trust_domain="condor_token" type="auth_file"/>
      </credentials>

A trust domain ending with '_token' signals the factory to handle tokens that are being forwarded. The factory will rename the forwarded token to the security_class, in this case 'frontend_token.

On the factory in the glideinWMS.xml config file, the security_class section must have matching names

            <security_classes>
               <security_class name="frontend" username="frontend"/>
               <security_class name="frontend_token" username="frontend" comment="_token in name signals special handling in creds dir"/>
            </security_classes>

The glidein entry points use the same trust_domain as the frontend, ending with '_token' i.e. 'condor_token'. The factory uses grid_proxy authentication to talk to the CE, but on seeing the name of the trust_domain it passes the frontend_token on to the CE which will be used for communication betweend the CE schedd/startd and the frontend Collector

     <entry name="el7_osg34" auth_method="grid_proxy" enabled="True" trust_domain="condor_token" gatekeeper="fermicloud378.fnal.gov fermicloud378.fnal.gov:9619" gridtype="condor" rsl="(queue=default)(jobtype=single)" schedd_name="fermicloud173.fnal.gov" verbosity="fast" work_dir="OSG">

To reassure that token authentication is being used between the CE and the frontend collector, attrs were configured to become part of the glideins CONDOR_CONFIG file

       <attr name="SEC_DEFAULT_AUTHENTICATION_METHODS" value="TOKEN" comment="glidein condor_config setting, use TOKEN auth exclusively" glidein_publish="True" job_publish="True" parameter="True" type="expr" />
       <attr name="ALL_DEBUG" value="D_SECURITY, D_FULLDEBUG" glidein_publish="True" job_publish="True" parameter="True" type="expr" />
        <attr name="SEC_TOKEN_DIRECTORY" value="$(LOCAL_DIR)/usertokens.d" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" />
        <attr name="SEC_TOKEN_SYSTEM_DIRECTORY" value="$(LOCAL_DIR)/systokens.d" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" />
        <attr name="TRUST_DOMAIN" comment="condor setting, not the same as trust_domain in frontend/factory xml"  glidein_publish="True" job_publish="True" parameter="True" type="expr" value="fermicloud115.fnal.gov:9618"/>

#10 Updated by Dennis Box about 1 month ago

attached full xml config files for frontend/factory

#11 Updated by Marco Mambelli 23 days ago

Some notes from the 10/22 meeting w/ HTCondor

Fixed in 8.9.4
CCB required daemon auth
- if it is only advertise startd cannot talk to the CCB
- workaround in the ticket
meeting w/ B.Bockelman and Todd - HTCondor token auth

Session coming from a tool is short-lived, for daemons it is days or weeks
Defaults to one day
There is a knob not documented, one for tools, one for daemons, to change the duration.
sec_<auth_level>_....
sec_client_session_duration
no one for all session duration

The duration is important only to get the job started, then there is the session duration.
But the session duration may be too short (1 hour), so the ticket is still needed to refresh that.

Session duration suggestion
- long term jobs
- negotiated and non negotiated sessions
- non negotiated security session are set to infinity by default
- if a session is not used for one hour it will automatically expire (startd session could expire, if there are no new job starting-stopping)

Thinking about mechanisms to renew or extend the token.
There is no mechanism to refresh input file
you could pull a file w/ chirp
keep a 2-week long token

Other suggestions/changes
1. make per-entry token to have better granularity
2. blacklisting token:
- creating token from a password, have multiple passwords, blacklist tokens w/ a specific password
- token has an identity, blacklist that identity

Currently tokens are not mapped, the identity is inside the token

There is python binding for request token
(not at the moment for create token)



Also available in: Atom PDF