This page documents running condor, including CredMon, inside a docker container on fermicloud, and authenticating with WLCG tokens which are similar to SciTokens.

Configure WLCG IAM

Register yourself using your email address at It may take a day or two to approve your registration so take this into account.

After your wlcg registration has completed, log in to its main web page. Click on the left-hand column entry 'Self Service Client Registration' Then click on the 'Register a new client +New Client ' button.

Make the 'client name' in this form something like 'scitoken-demo'

The 'redirect uris' field should be https://fermicloud-node-you-are-using/return/scitokens. In the picture shows doing this on fermicloud176.
Save your changes, the web page should look something like this:

Click on the 'Access' tab of the above page. We are going to make a new JWT scope for the tokens issued from this server. Add a 'condor' scope to the textbox lableled 'Scope' and save the page. This new scope is not exactly the same syntax as documented by the Scitokens and HTCondor sites, but it is similar. When I tried what I thought was the exact syntax things failed. This 'condor' setting in the scopes works for this example.

Save the 'Client ID' 'Client Secret' and 'Registration Access Token' fields, you will need them to configure the CredMon

Configure CredMon

Clone the htcondor Credmon repository

git clone

Change directories into the scitokens-credmon repository you just created. Ksu as you need copies of your hostcert and hostkey in the 'docker' subdirectory. Do not forget to remove the hostkey.pem file after the build.

[root@fermicloud176 scitokens-credmon]# cp /etc/grid-security/hostcert.pem docker
[root@fermicloud176 scitokens-credmon]# cp /etc/grid-security/hostkey.pem docker
[root@fermicloud176 scitokens-credmon]# chown dbox docker/*.pem

If you have never set up Docker on a fermicloud node see DockerOnFermicloud

Now as a non root user use the Client ID and Client Secrets you generated to build a container:

[dbox@fermicloud176 scitokens-credmon]$ export
[dbox@fermicloud176 scitokens-credmon]$ export SCI_ID=75247e08-afb1-47a3-983b-fe34e357d3b8
[dbox@fermicloud176 scitokens-credmon]$ export SCI_SECRET=AN98IUg-AAUbyvD4wPKYZuc7hXHqB0Fk9dwAU3IZOkojPYIPFl-DQvP2CA7DsJ_A19a39iJPWrj8kFVTLqlIpmo
[dbox@fermicloud176 scitokens-credmon]$ docker build \
>   --build-arg SCITOKENS_CLIENT_ID=${SCI_ID} \
>   --build-arg SCITOKENS_AUTHORIZATION_URL=https://${SERVER}:443/authorize \
>   --build-arg SCITOKENS_TOKEN_URL=https://${SERVER}:443/token \
>   --build-arg SCITOKENS_USER_URL=https://${SERVER}:443/userinfo \
>   --rm -t scitokens/htcondor-submit .

Go get a cup of coffee, the build takes a while.

When the build has completed, edit the docker-compose.yml file. Change the last 2 lines to reflect the fermicloud node you are running on

[dbox@fermicloud176 scitokens-credmon]$ tail -2 docker-compose.yml
    hostname: fermicloud176
[dbox@fermicloud176 scitokens-credmon]$

Run Credmon, Submit Condor Job

Now run the container, and log in to it!

[dbox@fermicloud176 scitokens-credmon]$ docker-compose up -d
Creating network "scitokens-credmon_default" with the default driver
Creating scitokens-credmon_scitokens-htcondor_1 ... done
[dbox@fermicloud176 scitokens-credmon]$ docker exec -it scitokens-credmon_scitokens-htcondor_1 /bin/su -l submitter
[submitter@fermicloud176 ~]$ ls -la
total 28
drwx------. 1 submitter submitter 4096 Feb 14 23:46 .
drwxr-xr-x. 1 root      root      4096 Feb 14 23:44 ..
-rw-r--r--. 1 submitter submitter   18 Aug  8  2019 .bash_logout
-rw-r--r--. 1 submitter submitter  193 Aug  8  2019 .bash_profile
-rw-r--r--. 1 submitter submitter  231 Aug  8  2019 .bashrc
-rwxr-xr-x. 1 submitter submitter  280 Feb 14 22:42
-rw-r--r--. 1 submitter submitter  314 Feb 14 22:42 test.sub
[submitter@fermicloud176 ~]$ cat test.sub
universe = vanilla

output = scitokens_test.$(cluster).$(process).out
error = scitokens_test.$(cluster).$(process).err
log = scitokens_test.$(cluster).log

executable =

use_oauth_services = scitokens

scitokens_oauth_permissions = read:/public
scitokens_oauth_resource = requested.resource.address


Edit test.sub to change scitokens_oauth_permissions to the scope we made earlier. In other words, change 'read:/public' to 'condor'.

After editing, you can submit the job!

[submitter@fermicloud176 ~]$ condor_submit test.sub
Submitting job(s)
Hello, submitter.
Please visit:

[submitter@fermicloud176 ~]$

When you push the login button, you are redirected back to the wlcg site to manually approve the token.

Once you approve the token, you are redirected back to the CredMon page

And, we can submit

[submitter@fermicloud176 ~]$ condor_submit test.sub
Submitting job(s).
1 job(s) submitted to cluster 1.

Useful Links and Reference

Condor Settings in the Container

[submitter@fermicloud176 ~]$ condor_config_val -dump | grep TOKEN
SCITOKENS_CLIENT_ID = 75247e08-afb1-47a3-983b-fe34e357d3b8
SCITOKENS_CLIENT_SECRET_FILE = /etc/condor/.secrets/scitokens
SEC_TOKEN_SYSTEM_DIRECTORY = /etc/condor/tokens.d
[submitter@fermicloud176 ~]$

WLCG Token Documentation

Indigo IAM Presentation