Project

General

Profile

SHAREDJOBSUBSETUP » History » Version 2

Version 1 (Katherine Lato, 06/03/2014 09:55 AM) → Version 2/10 (Arthur Kreymer, 10/02/2014 12:39 PM)

h1. SHAREDJOBSUBSETUP

h1. Setting up keytabs and proxies for jobsub in shared accounts

In certain cases we use shared accounts for production tasks:
* production grid job submission
* management of SAM metadata
* data handling tasks

Because Fermigrid jobs and other tasks require Grid proxies tied to an individual,
we need to make that individual's proxies available to the shared account.

The recommended technique is to copy a designated individual's keytab file into the shared account.
We place it in the path normally used by kcron, so that scripts can be written to run in both shared and individual accounts.
The method of generating and copying the keytab seems arcane and complicated.
There are reasons for each of the steps, so do everything described here !

*In the following discussion we describe copying the kreymer keytab to minospro on gpsn01.fnal.gov .*

We want to simply copy the kreymer kcron keytab file into the corresponding place in minospro.
* The name of each account's kcron keytab file under /var/adm/krb5 is given by kcron -f .
* File permissions prohibit the direct creation of the minospro keytab file.
* We run kcroninit under the minospro account to create a kcron keytab file.
* This creates a new kreymer keytab, which invalidates the former kreymer keytab on this host.
* So we run kcroninit in the kreymer account, which invalidates the keytab under minospro
* So we copy the new kreymer account keytab over the now stale keytab in minospro.

# kreymer opens two windows
** *_MINOSPRO_* ssh minospro@gpsn01.fnal.gov
** *_KREYMER_* ssh kreymer@gpsn01.fnal.gov
# *_MINOSPRO_*
** Create a kcron keytab with kreymer's kcron identity.
<pre>
kcroninit
</pre>
_(info) [we do this so that the kcron keytab file for MINOSPRO gets created with the right permissions and we can update it later]_
# *_KREYMER_*
** Make a fresh kreymer kcron keytab
** It is best to clear out the existing kcron keytab before making a new one.
<pre>
kcrondestroy
kcroninit
</pre>
_(info)[We do this because running kcroninit for our kerberos principal in the other account invalidated this copy]_
# *_KREYMER_*
** Verify that the keytab works with kcron
<pre>
kcron klist

Ticket cache: FILE:/tmp/krb5cc_1060_IVnxb30674
Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV

Valid starting Expires Service principal
10/11/13 09:09:09 10/11/13 19:09:09 krbtgt/FNAL.GOV@FNAL.GOV
...
</pre>
The important bits here are that you
#** don't get an error message
#** the Default principal: line has your username/cron/hostname in it
#** the ticket expiration date is in the future
# *_KREYMER_*
** Generate a command to be entered in the *_MINOSPRO_* window
<pre>
echo KEYUSE=/var/adm/krb5/`kcron -f`
KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
</pre>
_(info) [Here we're locating your kcron keytab file]_
# *_MINOSPRO_*
** enter the KEYUSE definition command echoed above
<pre>
KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
</pre>
# *_MINOSPRO_* Copy the kreymer key on top of the now obsolete minospro keytab.
<pre>
KEYSHARE=/var/adm/krb5/`kcron -f`
scp kreymer@gpsn01.fnal.gov:${KEYUSE} ${KEYSHARE}
</pre>
_(info) [Here we locate MINOSPRO's keytab, and copy ours over it. We use scp rather than cp so we have permission to do so]_
# *_MINOSPRO_*
** Test the keytab, see that we have kreymer's cron principal
<pre>
KEYTAB=/var/adm/krb5/`kcron -f`
KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d /`
kinit -5 -A -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
klist -f
Ticket cache: FILE:/tmp/krb5cc_42411_ZneQA30945
Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV

Valid starting Expires Service principal
10/11/13 09:13:56 10/11/13 19:13:56 krbtgt/FNAL.GOV@FNAL.GOV
...
</pre>
Once again, we are checking that
#** we don't get an error message
#** the Default principal: line has your username/cron/hostname in it
#** the ticket expiration date is in the future
_(info) [Here we're using kinit instead of kcron, because kcron will only look for MINOSPRO/cron/...@FNAL.GOV and not kreymer/cron/...@FNAL.GOV]_
# *_MINOSPRO_*
** Verify that we can get a production proxy for grid submission
<pre>
/scratch/grid/kproxy minos Production #
/scratch/grid/kproxy -i
...
lrwxrwxrwx 1 minospro gpcf 65 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
-rw------- 1 minospro gpcf 7004 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
lrwxrwxrwx 1 minospro gpcf 54 Oct 11 09:14 /scratch/minospro/grid/minospro.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy
...
/scratch/minospro/grid/minospro.minos.production.proxy
attribute : /fermilab/minos/Role=Production/Capability=NULL
Valid proxy expires in 43198 seconds (11 hours)
Valid proxy expires at Fri Oct 11 21:14:42 CDT 2013

</pre>
# *_MINOSPRO_*
** use cron to automate the process --
Run @crontab -e@ to edit the crontab for your shared account, and add lines:
<pre>
1 1-23/2 * * * /scratch/grid/kproxy minos Production
1 1-23/2 * * * /scratch/grid/kproxy minos Analysis
</pre>
(one for each role needed in the account)

You should now have a proxy refreshed automatically.

# * JOBSUB CLIENT *

When using the new jobsub_client in 2014, you do not run kproxy.
Instead, you need to have an active kerberos ticket.

Get this when needed withing your cron scripts with
<pre>
export KRB5CCNAME="FILE:/tmp/krb5cc_jobsub_`whoami`_${$}"
KEYTAB=/var/adm/krb5/`kcron -f`
KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d /`
kinit -5 -A -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV

</pre>