CredMonScitokensDocker¶
- Table of contents
- CredMonScitokensDocker
This page documents running condor, including CredMon, inside a docker container on fermicloud, and authenticating with WLCG tokens which are similar to SciTokens.
Configure WLCG IAM¶
Register yourself using your fnal.gov email address at https://wlcg.cloud.cnaf.infn.it. It may take a day or two to approve your registration so take this into account.
After your wlcg registration has completed, log in to its main web page. Click on the left-hand column entry 'Self Service Client Registration' Then click on the 'Register a new client +New Client ' button. 
Make the 'client name' in this form something like 'scitoken-demo'
The 'redirect uris' field should be https://fermicloud-node-you-are-using/return/scitokens. In the picture shows doing this on fermicloud176.
Save your changes, the web page should look something like this:
Click on the 'Access' tab of the above page. We are going to make a new JWT scope for the tokens issued from this server. Add a 'condor' scope to the textbox lableled 'Scope' and save the page. This new scope is not exactly the same syntax as documented by the Scitokens and HTCondor sites, but it is similar. When I tried what I thought was the exact syntax things failed. This 'condor' setting in the scopes works for this example.
Save the 'Client ID' 'Client Secret' and 'Registration Access Token' fields, you will need them to configure the CredMon
Configure CredMon¶
Clone the htcondor Credmon repository
git clone https://github.com/htcondor/scitokens-credmon.git
Change directories into the scitokens-credmon repository you just created. Ksu as you need copies of your hostcert and hostkey in the 'docker' subdirectory. Do not forget to remove the hostkey.pem file after the build.
[root@fermicloud176 scitokens-credmon]# cp /etc/grid-security/hostcert.pem docker [root@fermicloud176 scitokens-credmon]# cp /etc/grid-security/hostkey.pem docker [root@fermicloud176 scitokens-credmon]# chown dbox docker/*.pem
If you have never set up Docker on a fermicloud node see DockerOnFermicloud
Now as a non root user use the Client ID and Client Secrets you generated to build a container:
[dbox@fermicloud176 scitokens-credmon]$ export SERVER=wlcg.cloud.cnaf.infn.it
[dbox@fermicloud176 scitokens-credmon]$ export SCI_ID=75247e08-afb1-47a3-983b-fe34e357d3b8
[dbox@fermicloud176 scitokens-credmon]$ export SCI_SECRET=AN98IUg-AAUbyvD4wPKYZuc7hXHqB0Fk9dwAU3IZOkojPYIPFl-DQvP2CA7DsJ_A19a39iJPWrj8kFVTLqlIpmo
[dbox@fermicloud176 scitokens-credmon]$ docker build \
> --build-arg SCITOKENS_CLIENT_ID=${SCI_ID} \
> --build-arg SCITOKENS_CLIENT_SECRET=${SCI_SECRET} \
> --build-arg SCITOKENS_AUTHORIZATION_URL=https://${SERVER}:443/authorize \
> --build-arg SCITOKENS_TOKEN_URL=https://${SERVER}:443/token \
> --build-arg SCITOKENS_USER_URL=https://${SERVER}:443/userinfo \
> --rm -t scitokens/htcondor-submit .
Go get a cup of coffee, the build takes a while.
When the build has completed, edit the docker-compose.yml file. Change the last 2 lines to reflect the fermicloud node you are running on
[dbox@fermicloud176 scitokens-credmon]$ tail -2 docker-compose.yml
hostname: fermicloud176
domainname: fnal.gov
[dbox@fermicloud176 scitokens-credmon]$
Run Credmon, Submit Condor Job¶
Now run the container, and log in to it!
[dbox@fermicloud176 scitokens-credmon]$ docker-compose up -d Creating network "scitokens-credmon_default" with the default driver Creating scitokens-credmon_scitokens-htcondor_1 ... done [dbox@fermicloud176 scitokens-credmon]$ docker exec -it scitokens-credmon_scitokens-htcondor_1 /bin/su -l submitter [submitter@fermicloud176 ~]$ ls -la total 28 drwx------. 1 submitter submitter 4096 Feb 14 23:46 . drwxr-xr-x. 1 root root 4096 Feb 14 23:44 .. -rw-r--r--. 1 submitter submitter 18 Aug 8 2019 .bash_logout -rw-r--r--. 1 submitter submitter 193 Aug 8 2019 .bash_profile -rw-r--r--. 1 submitter submitter 231 Aug 8 2019 .bashrc -rwxr-xr-x. 1 submitter submitter 280 Feb 14 22:42 test.sh -rw-r--r--. 1 submitter submitter 314 Feb 14 22:42 test.sub [submitter@fermicloud176 ~]$ cat test.sub universe = vanilla output = scitokens_test.$(cluster).$(process).out error = scitokens_test.$(cluster).$(process).err log = scitokens_test.$(cluster).log executable = test.sh use_oauth_services = scitokens scitokens_oauth_permissions = read:/public scitokens_oauth_resource = requested.resource.address queue
Edit test.sub to change scitokens_oauth_permissions to the scope we made earlier. In other words, change 'read:/public' to 'condor'.
After editing, you can submit the job!
[submitter@fermicloud176 ~]$ condor_submit test.sub Submitting job(s) Hello, submitter. Please visit: https://fermicloud176.fnal.gov/key/26b2e9ea909ffd36f92050d8ad662204bdd11635a8217db563a3795c2b37715c [submitter@fermicloud176 ~]$
When you push the login button, you are redirected back to the wlcg site to manually approve the token.
Once you approve the token, you are redirected back to the CredMon page
And, we can submit
[submitter@fermicloud176 ~]$ condor_submit test.sub Submitting job(s). 1 job(s) submitted to cluster 1.
Useful Links and Reference¶
Condor Settings in the Container¶
[submitter@fermicloud176 ~]$ condor_config_val -dump | grep TOKEN SCITOKENS_AUTHORIZATION_URL = https://wlcg.cloud.cnaf.infn.it:443/authorize SCITOKENS_CLIENT_ID = 75247e08-afb1-47a3-983b-fe34e357d3b8 SCITOKENS_CLIENT_SECRET_FILE = /etc/condor/.secrets/scitokens SCITOKENS_RETURN_URL_SUFFIX = /return/scitokens SCITOKENS_TOKEN_URL = https://wlcg.cloud.cnaf.infn.it:443/token SCITOKENS_USER_URL = https://wlcg.cloud.cnaf.infn.it:443/userinfo SEC_ENABLE_IMPERSONATION_TOKENS = false SEC_IMPERSONATION_TOKEN_LIMITS = SEC_ISSUED_TOKEN_EXPIRATION = SEC_TOKEN_DIRECTORY = SEC_TOKEN_ISSUER_KEY = POOL SEC_TOKEN_MAX_AGE = SEC_TOKEN_SYSTEM_DIRECTORY = /etc/condor/tokens.d [submitter@fermicloud176 ~]$
WLCG Token Documentation¶
https://docs.google.com/document/d/1cNm4nBl9ELhExwLxswpxLLNTuz8pT38-b_DewEyEWug