Project

General

Profile

SHAREDJOBSUBSETUP » History » Version 4

Arthur Kreymer, 10/02/2014 12:42 PM
fixed typo

1 1 Katherine Lato
h1. SHAREDJOBSUBSETUP
2 1 Katherine Lato
3 1 Katherine Lato
h1. Setting up keytabs and proxies for jobsub in shared accounts
4 1 Katherine Lato
5 1 Katherine Lato
In certain cases we use shared accounts for production tasks:
6 1 Katherine Lato
* production grid job submission
7 1 Katherine Lato
* management of SAM metadata
8 1 Katherine Lato
* data handling tasks
9 1 Katherine Lato
10 1 Katherine Lato
Because Fermigrid jobs and other tasks require Grid proxies tied to an individual,
11 1 Katherine Lato
we need to make that individual's proxies available to the shared account.
12 1 Katherine Lato
13 1 Katherine Lato
The recommended technique is to copy a designated individual's keytab file into the shared account.
14 1 Katherine Lato
We place it in the path normally used by kcron, so that scripts can be written to run in both shared and individual accounts.
15 1 Katherine Lato
The method of generating and copying the keytab seems arcane and complicated.
16 1 Katherine Lato
There are reasons for each of the steps, so do everything described here !
17 1 Katherine Lato
18 1 Katherine Lato
*In the following discussion we describe copying the kreymer keytab to minospro on gpsn01.fnal.gov .*
19 1 Katherine Lato
20 1 Katherine Lato
We want to simply copy the kreymer kcron keytab file into the corresponding place in minospro.
21 1 Katherine Lato
* The name of each account's kcron keytab file under /var/adm/krb5 is given by   kcron -f .
22 1 Katherine Lato
* File permissions prohibit the direct creation of the minospro keytab file.
23 1 Katherine Lato
* We run kcroninit under the minospro account to create a kcron keytab file.
24 1 Katherine Lato
* This creates a new kreymer keytab, which invalidates the former kreymer keytab on this host.
25 1 Katherine Lato
* So we run kcroninit in the kreymer account, which invalidates the keytab under minospro
26 1 Katherine Lato
* So we copy the new kreymer account keytab over the now stale keytab in minospro.
27 1 Katherine Lato
28 1 Katherine Lato
# kreymer opens two windows 
29 1 Katherine Lato
** *_MINOSPRO_* ssh minospro@gpsn01.fnal.gov
30 1 Katherine Lato
** *_KREYMER_* ssh kreymer@gpsn01.fnal.gov
31 1 Katherine Lato
# *_MINOSPRO_*
32 1 Katherine Lato
** Create a kcron keytab with kreymer's kcron identity.
33 1 Katherine Lato
<pre>
34 1 Katherine Lato
kcroninit
35 1 Katherine Lato
</pre>
36 1 Katherine Lato
_(info) [we do this so that the kcron keytab file for MINOSPRO gets created with the right permissions and we can update it later]_
37 1 Katherine Lato
# *_KREYMER_*
38 1 Katherine Lato
** Make a fresh kreymer kcron keytab
39 1 Katherine Lato
** It is best to clear out the existing kcron keytab before making a new one.
40 1 Katherine Lato
<pre>
41 1 Katherine Lato
kcrondestroy
42 1 Katherine Lato
kcroninit
43 1 Katherine Lato
</pre>
44 1 Katherine Lato
_(info)[We do this because running kcroninit for our kerberos principal in the other account invalidated this copy]_
45 1 Katherine Lato
# *_KREYMER_*
46 1 Katherine Lato
** Verify that the keytab works with kcron
47 1 Katherine Lato
<pre>
48 1 Katherine Lato
kcron klist
49 1 Katherine Lato
50 1 Katherine Lato
Ticket cache: FILE:/tmp/krb5cc_1060_IVnxb30674
51 1 Katherine Lato
Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
52 1 Katherine Lato
53 1 Katherine Lato
Valid starting     Expires            Service principal
54 1 Katherine Lato
10/11/13 09:09:09  10/11/13 19:09:09  krbtgt/FNAL.GOV@FNAL.GOV
55 1 Katherine Lato
...
56 1 Katherine Lato
</pre>
57 1 Katherine Lato
The important bits here are that you
58 1 Katherine Lato
#** don't get an error message
59 1 Katherine Lato
#** the Default principal: line has your username/cron/hostname in it
60 1 Katherine Lato
#** the ticket expiration date is in the future
61 1 Katherine Lato
# *_KREYMER_*
62 1 Katherine Lato
** Generate a command to be entered in the  *_MINOSPRO_* window
63 1 Katherine Lato
<pre>
64 1 Katherine Lato
echo KEYUSE=/var/adm/krb5/`kcron -f`
65 1 Katherine Lato
KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
66 1 Katherine Lato
</pre>
67 1 Katherine Lato
_(info) [Here we're locating your kcron keytab file]_
68 1 Katherine Lato
# *_MINOSPRO_* 
69 1 Katherine Lato
** enter the KEYUSE definition command echoed above
70 1 Katherine Lato
<pre>
71 1 Katherine Lato
KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
72 1 Katherine Lato
</pre>
73 1 Katherine Lato
# *_MINOSPRO_* Copy the kreymer key on top of the now obsolete minospro keytab.
74 1 Katherine Lato
<pre>
75 1 Katherine Lato
KEYSHARE=/var/adm/krb5/`kcron -f`
76 1 Katherine Lato
scp kreymer@gpsn01.fnal.gov:${KEYUSE} ${KEYSHARE}
77 1 Katherine Lato
</pre>
78 1 Katherine Lato
_(info) [Here we locate MINOSPRO's keytab, and copy ours over it. We use scp rather than cp so we have permission to do so]_
79 1 Katherine Lato
# *_MINOSPRO_* 
80 1 Katherine Lato
** Test the keytab, see that we have kreymer's cron principal
81 1 Katherine Lato
<pre>
82 1 Katherine Lato
KEYTAB=/var/adm/krb5/`kcron -f`
83 1 Katherine Lato
KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d /`
84 1 Katherine Lato
kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
85 1 Katherine Lato
klist -f
86 1 Katherine Lato
Ticket cache: FILE:/tmp/krb5cc_42411_ZneQA30945
87 1 Katherine Lato
Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
88 1 Katherine Lato
89 1 Katherine Lato
Valid starting     Expires            Service principal
90 1 Katherine Lato
10/11/13 09:13:56  10/11/13 19:13:56  krbtgt/FNAL.GOV@FNAL.GOV
91 1 Katherine Lato
...
92 1 Katherine Lato
</pre>
93 1 Katherine Lato
Once again, we are checking that
94 1 Katherine Lato
#** we don't get an error message
95 1 Katherine Lato
#** the Default principal: line has your username/cron/hostname in it
96 1 Katherine Lato
#** the ticket expiration date is in the future
97 1 Katherine Lato
_(info) [Here we're using kinit instead of kcron, because kcron will only look for MINOSPRO/cron/...@FNAL.GOV and not kreymer/cron/...@FNAL.GOV]_
98 1 Katherine Lato
# *_MINOSPRO_* 
99 1 Katherine Lato
** Verify that we can get a production proxy for grid submission
100 1 Katherine Lato
<pre>
101 1 Katherine Lato
/scratch/grid/kproxy minos Production  # 
102 1 Katherine Lato
/scratch/grid/kproxy -i
103 1 Katherine Lato
...
104 1 Katherine Lato
lrwxrwxrwx 1 minospro gpcf   65 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
105 1 Katherine Lato
-rw------- 1 minospro gpcf 7004 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
106 1 Katherine Lato
lrwxrwxrwx 1 minospro gpcf   54 Oct 11 09:14 /scratch/minospro/grid/minospro.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy
107 1 Katherine Lato
...
108 1 Katherine Lato
/scratch/minospro/grid/minospro.minos.production.proxy
109 1 Katherine Lato
attribute : /fermilab/minos/Role=Production/Capability=NULL
110 1 Katherine Lato
Valid proxy expires in 43198 seconds (11 hours)
111 1 Katherine Lato
Valid proxy expires at Fri Oct 11 21:14:42 CDT 2013
112 1 Katherine Lato
113 1 Katherine Lato
114 1 Katherine Lato
</pre>
115 1 Katherine Lato
# *_MINOSPRO_* 
116 1 Katherine Lato
** use cron to automate the process -- 
117 1 Katherine Lato
Run @crontab -e@ to edit the crontab for  your shared account, and add lines:
118 1 Katherine Lato
<pre>
119 1 Katherine Lato
1 1-23/2 * * * /scratch/grid/kproxy minos Production
120 1 Katherine Lato
1 1-23/2 * * * /scratch/grid/kproxy minos Analysis
121 1 Katherine Lato
</pre>
122 1 Katherine Lato
(one for each role needed in the account) 
123 1 Katherine Lato
You should now have a proxy refreshed automatically.
124 4 Arthur Kreymer
# *_JOBSUB CLIENT_*
125 2 Arthur Kreymer
When using the new jobsub_client in 2014, you do not run kproxy.
126 2 Arthur Kreymer
Instead, you need to have an active kerberos ticket.
127 3 Arthur Kreymer
Get this when needed within your cron scripts with
128 2 Arthur Kreymer
<pre>
129 2 Arthur Kreymer
export KRB5CCNAME="FILE:/tmp/krb5cc_jobsub_`whoami`_${$}"
130 2 Arthur Kreymer
KEYTAB=/var/adm/krb5/`kcron -f`
131 2 Arthur Kreymer
KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d /`
132 2 Arthur Kreymer
kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
133 2 Arthur Kreymer
134 2 Arthur Kreymer
</pre>