Feature #23092

Use token-auth for Glideins authentication and add support for sci-token

Added by Marco Mambelli about 1 month ago. Updated 1 day ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)




OSG plans to drop support for x509 by January 2020
GlideinWMS should start using HTCondor's token-auth for its internal authentication (the Frontend requests a token to the Collector and forwards it to the Glidein via Factory)
Support for sci-token should be added to GlideinWMS
This development should happen in the 3.5 series

Marco had a meeting w/ Brian Bockelman and Brian Lin. Attached are some notes.
Here the action Items that will drive this ticket (GWMS refers to a GlideinWMS developer - the owner of this ticket):
  • OSG will make available an RPM of HTCondor 8.9.2 with SciTokens ASAP.
  • In the meantime, GWMS will take 8.9.2 from without SciTokens.
  • GWMS can immediately start working on TOKEN auth completely internal to gWMS.
  • We think this is doable by early September.
  • GWMS and Derek Weitzel will work to help GWMS generate a public/private keypair and post the public portion to the issuer.
    • Also Derek will help with the CLI to generate tokens with the goal of having gWMS use the scitokens python API directly eventually.
    • Once this is done, GWMS can generate SciTokens but has no CE to submit them to. Everything can be tested but the final "submit pilot to CE" step.
  • OSG will aim to provide a CE that accepts the dteam SciTokens near the end of August.
    • Goal is to have a CE package in osg-development by early September.
  • At least GWMS, Brian, and Brian will aim to have a "hackathon" day at FNAL in early October for this project to either finish late deliverables or put together a demo.

Here the link for the HTCondor development repo (for 8.9.2):

tokenauth-notes1.pdf (345 KB) tokenauth-notes1.pdf Marco Mambelli, 08/09/2019 02:04 AM
tokenauth-notes2.pdf (691 KB) tokenauth-notes2.pdf Marco Mambelli, 08/09/2019 02:04 AM


Support #23278: Condor 8.9.2 configuration changesNewDennis Box


#1 Updated by Dennis Box 30 days ago

I realized this ticket is where some of my work notes should go so they get preserved.

To upgrade a working factory/frontend to condor 8.9.2

  1. yum install --enablerepo osg-upcoming-development condor
  2. add this to condor config ( /etc/condor/config.d/03_gwms_local.config)
    For Frontend:
    ALLOW_READ = *
    For Factory:
    ALLOW_READ = *
  3. systemctl restart condor; systemctl reload gwms(factory, frontend) <==== should now work with condor 8.9.2 and GSI authentication

Generating a Condor Token

  • Tokens design document
  • JWT (Json Web Tokens) RFC
  • To generate a condor token as root:
    • condor_store_cred -f /etc/condor/passwords.d/password
    • set SEC_PASSWORD_FILE=/etc/condor/passwords.d/password; condor_reconfig
    • condor_token_create -identity (example: condor@$HOSTNAME) -lifetime (seconds) -authz (auth_options) > tokenfile
  • My task (as I currently understand it) is now to generate condor tokens with the correct identities and authz options to allow the Frontend collector to authenticate with the startds in the glideins on the CE.

#2 Updated by Dennis Box 30 days ago

Generating SciTokens

  • Getting SciTokens to work with condor/glideinwms is a step that happens after I get Condor Tokens to work, but generating a SciToken is not hard:
  1. yum install python2-scitokens
  2. generate some keys
    scitokens-admin-create-key --create-keys --pem-private > /tmp/test.scitoken.private.pem
    scitokens-admin-create-key --private-keyfile /tmp/test.scitoken.private.pem --jwks-private > /tmp/test.scitoken.private.jwks
    scitokens-admin-create-key --private-keyfile /tmp/test.scitoken.private.pem --jwks-public > /tmp/test.scitoken.public.jwks
  3. copy the keys over to a web server (say for sake of argument
    cd /var/www/html
    mkdir -p oauth2/certs/.well-known
    create file oauth2/certs/.well-known/openid-configuration with this content:


    copy test.scitoken.public.jwks to /oauth2/certs directory

  4. Generate a token
    scitokens-admin-create-token --key_id 1d92 --keyfile /tmp/test.scitoken.private.pem --issuer sub=htcondor 'scope=condor:/READ condor:/WRITE condor:/ALLOW' > condor.token

#3 Updated by Dennis Box 30 days ago

Condor Authentication with Tokens

  • TOKEN and SCITOKEN are both valid settings for SEC_DEFAULT_AUTHENTICATION_METHODS starting with 8.9.2
  • SEC_(DAEMON)_AUTHENTICATION_METHODS now exists for all the condor daemons, so we can customize authentication to a fine grained level if desired.
  • condor_token_create -authz input arguments are discussed here :, for example -authz ADVERTISE_STARTD

#4 Updated by Brian Lin 26 days ago

Dennis Box wrote:

Condor Authentication with Tokens

  • TOKEN and SCITOKEN are both valid settings for SEC_DEFAULT_AUTHENTICATION_METHODS starting with 8.9.2

I believe that unrecognized values in SEC_*_AUTHENTICATION_METHODS are just ignored so you shouldn't have to worry about using if statements in the HTCondor config.

#5 Updated by Dennis Box 18 days ago

User Authorization with Tokens for Job Submission

  • thus far, factory and frontend have had SEC_DEFAULT_AUTHENTICATION_METHODS = FS,GSI
  • change this setting to SEC_DEFAULT_AUTHENTICATION_METHODS = TOKEN,FS,GSI . Restart.
    • frontend, factory, and CEs still play together nicely, starting glideins after a user job submission
  • Add setting SEC_CLIENT_AUTHENTICATION_METHODS = TOKEN,GSI to frontend condor configuration
    • we now need a token to submit, as this 'helpful' error message informs us:
      [dbox@fermicloud096 testjobs]$ condor_submit testjob.singularity.jdf 
      Submitting job(s).
      ERROR: Failed to create proc
    • I have ALL_DEBUG = D_SECURITY D_COMMAND set, there is probably something in the log that matches this but I cant find it.
      The error is clearly because I need a token. After playing with the condor_token_ commands for a while this is how I finally enabled submission:
      + mkdir -p /cloud/login/dbox/.condor/tokens.d
      + touch /cloud/login/dbox/.condor/tokens.d/dbox.token
      + condor_token_create -identity >> /cloud/login/dbox/.condor/tokens.d/dbox.token
      + chmod 400 /cloud/login/dbox/.condor/tokens.d/dbox.token
      + chown -R dbox /cloud/login/dbox/.condor
  • this token is pretty 'wide open' , no expiration date or internal limit to what it is allowed to do. Obviously not suitable for a production environment. I am just trying to get things up and running, and this does allow submissions.
  • to get jobs to start I had to do this as root:
     condor_token_create -identity condor@$HOSTNAME        -authz ADMINISTRATOR > /etc/condor/tokens.d/admin.token
  • I think this token being 'wide open' is less of a problem than the user token, its somewhat like having an encrypted root password in /etc/passwd on unix. Not ideal but not horrifically bad like the user token.

Increasing the scope of daemons that use TOKEN instead of FS or GSI

  • In order to understand this new security model and TOKEN authentications place in it, I am tightening down the SEC_*_AUTHENTICATION_METHODS one by one until something breaks, then seeing what I need to do to fix it.
  • Turning these knobs 1 by 1 on the frontend still allows me a fully working gwms submission system with the above 2 (probably overly generous) tokens
    • etc through all the knobs, eliminating FS authentication.
    • end result is a working frontend that uses GSI and TOKEN authentication only, glideins start, phone home, user jobs run
  • Now start turning off GSI to see how far we can take this without further code modification

#6 Updated by Dennis Box 5 days ago

  • Start date changed from 08/09/2019 to 09/16/2019
  • Due date changed from 08/30/2019 to 09/16/2019

due to changes in a related task: #23278

#7 Updated by Marco Mambelli 4 days ago

  • Target version changed from v3_5_x to v3_7

#8 Updated by Dennis Box 1 day ago

Recipe for Sending Glideins That Connect Back With Token Auth

  • This is not 'production ready', it is more of a proof of concept
  • Requirements:
    • Glidein entry point runs glideins with condor 8.9.2
    • A token, osg.token has been created and placed in /etc/gwms-frontend/creds/osg.token
      • no work has (yet) been done on securing osg.token, its permissions are too broad for anything other than demonstration purposes
      • if osg.token has an expiration date, a mechanism must be in place to renew it periodically
  • add to the <attrs> section of frontend.xml
          <attr name="SEC_DEFAULT_AUTHENTICATION_METHODS" comment="glidein condor_config setting, use TOKEN auth exclusively" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="TOKEN"/>
          <attr name="SEC_TOKEN_DIRECTORY" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(LOCAL_DIR)/usertokens.d"/>
          <attr name="SEC_TOKEN_SYSTEM_DIRECTORY" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(LOCAL_DIR)/systokens.d"/>
          <attr name="SEC_PASSWORD_DIRECTORY" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(LOCAL_DIR)/passwords.d"/>
          <attr name="SEC_PASSWORD_FILE" comment="glidein condor setting" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="$(SEC_PASSWORD_DIRECTORY)/password"/>
          <attr name="TOKEN_FILE" comment="goes in SEC_TOKEN_DIRECTORY" glidein_publish="True" job_publish="True" parameter="True" type="expr" value="osg.token"/>
          <attr name="EXTRA_LIB_LIST" comment="libraries needed for token auth by condor 8.9.2" glidein_publish="True" job_publish="True" parameter="True" type="expr" value=""/>
  • add to <files> section of frontend.xml
                <file absfname="/usr/lib64/" comment="needed for TOKEN auth in glidein" after_entry="False" const="True" executable="False" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
                <file absfname="/usr/lib64/" comment="needed for TOKEN auth in glidein" after_entry="False" const="True" executable="False" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
                <file absfname="/etc/gwms-frontend/creds/osg.token" comment="actual token that is used for authorization" after_entry="False" const="True" executable="False" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
                <file absfname="/var/lib/gwms-frontend/web-base/frontend/" comment="script to create dirs and move uploaded files correct places in glidein" after_entry="False" const="True" executable="True" period="0" prefix="GLIDEIN_PS_" untar="False" wrapper="False">
                   <untar_options cond_attr="TRUE"/>
  • contents of, that creates needed directories and makes sure uploaded files are in correct place

glidein_config_val () {
   grep "^$1 " $glidein_config | cut -d ' ' -f 2-

if [ "$glidein_config" = "" ]; then

GWMS_THIS_SCRIPT="`basename "$0"`" 
GWMS_THIS_SCRIPT_DIR="`dirname "$0"`" 

EXTRA_LIB_LIST=$(glidein_config_val EXTRA_LIB_LIST)
TOKEN_FILE=$(glidein_config_val TOKEN_FILE)
CONDOR_DIR=$(glidein_config_val CONDOR_DIR)

SEC_TOKEN_DIRECTORY=$(echo $(glidein_config_val SEC_TOKEN_DIRECTORY) | sed -e 's/.*\///')

SEC_TOKEN_SYSTEM_DIRECTORY=$(echo $(glidein_config_val SEC_TOKEN_SYSTEM_DIRECTORY)| sed -e 's/.*\///')

SEC_PASSWORD_DIRECTORY=$(echo $(glidein_config_val SEC_PASSWORD_DIRECTORY) | sed -e 's/.*\///')

SEC_PASSWORD_FILE=$(echo $(glidein_config_val SEC_PASSWORD_FILE) | sed -e 's/.*\///')

for LIB in $EXTRA_LIB_LIST; do
cd ${CONDOR_DIR}/lib
ln -s
ln -s
ln -s
cd -

  • systemctl reload gwms-frontend and now glideins auth back to the frontend collector using TOKEN auth

Also available in: Atom PDF