Project

General

Profile

SHAREDJOBSUBSETUP » History » Version 10

Arthur Kreymer, 03/25/2015 11:41 AM
Added link to OSG Service Cert document, noted April 2015 planned support.

1 1 Katherine Lato
h1. SHAREDJOBSUBSETUP
2 1 Katherine Lato
3 1 Katherine Lato
h1. Setting up keytabs and proxies for jobsub in shared accounts
4 1 Katherine Lato
5 1 Katherine Lato
In certain cases we use shared accounts for production tasks:
6 1 Katherine Lato
* production grid job submission
7 1 Katherine Lato
* management of SAM metadata
8 1 Katherine Lato
* data handling tasks
9 1 Katherine Lato
10 1 Katherine Lato
Because Fermigrid jobs and other tasks require Grid proxies tied to an individual,
11 10 Arthur Kreymer
we need to make such proxies available to the shared account.
12 1 Katherine Lato
13 10 Arthur Kreymer
An OSG Service Certificates can be used to make such a proxy,
14 10 Arthur Kreymer
and will be supported by jobsub_client in early April 2015.
15 10 Arthur Kreymer
See support documentation at 
16 10 Arthur Kreymer
https://cdcvs.fnal.gov/redmine/projects/offline_production_operations_service/wiki/Configuring_shared_account_OSG_certificates
17 9 Arthur Kreymer
18 10 Arthur Kreymer
Meanwhile, the recommended technique is to copy a designated individual's keytab file into the shared account.
19 1 Katherine Lato
We place it in the path normally used by kcron, so that scripts can be written to run in both shared and individual accounts.
20 1 Katherine Lato
The method of generating and copying the keytab seems arcane and complicated.
21 1 Katherine Lato
There are reasons for each of the steps, so do everything described here !
22 1 Katherine Lato
23 1 Katherine Lato
*In the following discussion we describe copying the kreymer keytab to minospro on gpsn01.fnal.gov .*
24 1 Katherine Lato
25 10 Arthur Kreymer
*Pick a user who will not likely be running grid jobs, as production use will modify the users' priorities.*
26 10 Arthur Kreymer
27 10 Arthur Kreymer
*New users of this method are urged to wait for OSG Service Cert proxy support in April 2015*
28 5 Arthur Kreymer
29 1 Katherine Lato
We want to copy the kreymer kcron keytab file into the corresponding place in minospro.
30 1 Katherine Lato
* The name of each account's kcron keytab file under /var/adm/krb5 is given by   kcron -f .
31 5 Arthur Kreymer
* File permissions prohibit the direct creation of the minospro keytab file.
32 5 Arthur Kreymer
* Run kcroninit under the minospro account to create a minospro@FNAL.GOV keytab (give a fake password)
33 1 Katherine Lato
* Copy the kreymer account keytab into the minospro keytab file
34 1 Katherine Lato
35 1 Katherine Lato
# kreymer opens two windows 
36 1 Katherine Lato
** *_MINOSPRO_* ssh minospro@gpsn01.fnal.gov
37 1 Katherine Lato
** *_KREYMER_* ssh kreymer@gpsn01.fnal.gov
38 5 Arthur Kreymer
# *_MINOSPRO_*
39 1 Katherine Lato
** Create an invalid kcron keytab to be overwritten.
40 1 Katherine Lato
<pre>
41 5 Arthur Kreymer
kcroninit
42 5 Arthur Kreymer
...
43 5 Arthur Kreymer
Are you on a secure channel?  (default = y): y
44 5 Arthur Kreymer
What is your kerberos principal (default = kreymer@FNAL.GOV): 
45 1 Katherine Lato
Enter the password for minospro@FNAL.GOV: junq
46 1 Katherine Lato
</pre>
47 8 Arthur Kreymer
_(info) [we do this so that the kcron keytab file for MINOSPRO gets created with the right permissions and we can update it later]_
48 8 Arthur Kreymer
** Check that the keytab was created
49 8 Arthur Kreymer
<pre>
50 8 Arthur Kreymer
ls -l /var/adm/krb5/`kcron -f`
51 1 Katherine Lato
</pre>
52 5 Arthur Kreymer
# *_KREYMER_*
53 1 Katherine Lato
** Verify that the existing kreymer keytab works with kcron
54 1 Katherine Lato
<pre>
55 1 Katherine Lato
kcron klist
56 1 Katherine Lato
57 1 Katherine Lato
Ticket cache: FILE:/tmp/krb5cc_1060_IVnxb30674
58 1 Katherine Lato
Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
59 1 Katherine Lato
60 1 Katherine Lato
Valid starting     Expires            Service principal
61 1 Katherine Lato
10/11/13 09:09:09  10/11/13 19:09:09  krbtgt/FNAL.GOV@FNAL.GOV
62 1 Katherine Lato
...
63 1 Katherine Lato
</pre>
64 1 Katherine Lato
The important bits here are that you
65 1 Katherine Lato
#** don't get an error message
66 1 Katherine Lato
#** the Default principal: line has your username/cron/hostname in it
67 1 Katherine Lato
#** the ticket expiration date is in the future
68 1 Katherine Lato
# *_KREYMER_*
69 1 Katherine Lato
** Generate a command to be entered in the  *_MINOSPRO_* window
70 1 Katherine Lato
<pre>
71 1 Katherine Lato
echo KEYUSE=/var/adm/krb5/`kcron -f`
72 1 Katherine Lato
KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
73 1 Katherine Lato
</pre>
74 5 Arthur Kreymer
_(info) [Here we're locating your kcron keytab file]_
75 1 Katherine Lato
# *_MINOSPRO_* Copy the kreymer keytab into of the invalid minospro keytab.
76 1 Katherine Lato
<pre>
77 1 Katherine Lato
KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
78 1 Katherine Lato
KEYSHARE=/var/adm/krb5/`kcron -f`
79 1 Katherine Lato
scp kreymer@gpsn01.fnal.gov:${KEYUSE} ${KEYSHARE}
80 1 Katherine Lato
</pre>
81 1 Katherine Lato
_(info) [Here we locate MINOSPRO's keytab, and copy ours over it. We use scp rather than cp so we have permission to do so]_
82 1 Katherine Lato
# *_MINOSPRO_* 
83 1 Katherine Lato
** Test the keytab, see that we have kreymer's cron principal
84 1 Katherine Lato
<pre>
85 6 Arthur Kreymer
KEYTAB=/var/adm/krb5/`kcron -f`
86 1 Katherine Lato
KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d / | head -1`
87 1 Katherine Lato
kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
88 1 Katherine Lato
klist -f
89 1 Katherine Lato
Ticket cache: FILE:/tmp/krb5cc_42411_ZneQA30945
90 1 Katherine Lato
Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
91 1 Katherine Lato
92 1 Katherine Lato
Valid starting     Expires            Service principal
93 1 Katherine Lato
10/11/13 09:13:56  10/11/13 19:13:56  krbtgt/FNAL.GOV@FNAL.GOV
94 1 Katherine Lato
...
95 1 Katherine Lato
</pre>
96 1 Katherine Lato
Once again, we are checking that
97 1 Katherine Lato
#** we don't get an error message
98 1 Katherine Lato
#** the Default principal: line has your username/cron/hostname in it
99 1 Katherine Lato
#** the ticket expiration date is in the future
100 1 Katherine Lato
_(info) [Here we're using kinit instead of kcron, because kcron will only look for MINOSPRO/cron/...@FNAL.GOV and not kreymer/cron/...@FNAL.GOV]_
101 1 Katherine Lato
# *_MINOSPRO_* 
102 1 Katherine Lato
** Verify that we can get a production proxy for grid submission
103 1 Katherine Lato
<pre>
104 1 Katherine Lato
/scratch/grid/kproxy minos Production  # 
105 1 Katherine Lato
/scratch/grid/kproxy -i
106 1 Katherine Lato
...
107 1 Katherine Lato
lrwxrwxrwx 1 minospro gpcf   65 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
108 1 Katherine Lato
-rw------- 1 minospro gpcf 7004 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
109 1 Katherine Lato
lrwxrwxrwx 1 minospro gpcf   54 Oct 11 09:14 /scratch/minospro/grid/minospro.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy
110 1 Katherine Lato
...
111 1 Katherine Lato
/scratch/minospro/grid/minospro.minos.production.proxy
112 1 Katherine Lato
attribute : /fermilab/minos/Role=Production/Capability=NULL
113 1 Katherine Lato
Valid proxy expires in 43198 seconds (11 hours)
114 1 Katherine Lato
Valid proxy expires at Fri Oct 11 21:14:42 CDT 2013
115 1 Katherine Lato
116 1 Katherine Lato
117 1 Katherine Lato
</pre>
118 1 Katherine Lato
# *_MINOSPRO_* 
119 1 Katherine Lato
** use cron to automate the process -- 
120 1 Katherine Lato
Run @crontab -e@ to edit the crontab for  your shared account, and add lines:
121 1 Katherine Lato
<pre>
122 1 Katherine Lato
1 1-23/2 * * * /scratch/grid/kproxy minos Production
123 1 Katherine Lato
1 1-23/2 * * * /scratch/grid/kproxy minos Analysis
124 1 Katherine Lato
</pre>
125 1 Katherine Lato
(one for each role needed in the account) 
126 4 Arthur Kreymer
You should now have a proxy refreshed automatically.
127 2 Arthur Kreymer
# *_JOBSUB CLIENT_*
128 2 Arthur Kreymer
When using the new jobsub_client in 2014, you do not run kproxy.
129 3 Arthur Kreymer
Instead, you need to have an active kerberos ticket.
130 2 Arthur Kreymer
Get this when needed within your cron scripts with
131 2 Arthur Kreymer
<pre>
132 2 Arthur Kreymer
export KRB5CCNAME="FILE:/tmp/krb5cc_jobsub_`whoami`_${$}"
133 6 Arthur Kreymer
KEYTAB=/var/adm/krb5/`kcron -f`
134 2 Arthur Kreymer
KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d / | head -1`
135 2 Arthur Kreymer
kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
136 2 Arthur Kreymer
137 7 Arthur Kreymer
</pre>
138 7 Arthur Kreymer
139 7 Arthur Kreymer
h2. Using the shared account for samweb access
140 7 Arthur Kreymer
141 7 Arthur Kreymer
Using the same procedure to get a shared ticket,
142 7 Arthur Kreymer
we can also access samweb from a shared account.
143 7 Arthur Kreymer
144 7 Arthur Kreymer
# Register the shared idenity with samweb, mapping to an appropriate account
145 7 Arthur Kreymer
<pre>
146 7 Arthur Kreymer
https://samweb.fnal.gov:8483/sam/minos/admin/users
147 7 Arthur Kreymer
    Added the account
148 7 Arthur Kreymer
    Added an appropriate Grid subject to that account, like
149 7 Arthur Kreymer
/DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=minos27.fnal.gov/CN=cron/CN=Arthur E. Kreymer/CN=UID:kreymer
150 7 Arthur Kreymer
</pre>
151 7 Arthur Kreymer
# Get a ticket as with jobsub, with a unique ticket file
152 7 Arthur Kreymer
<pre>
153 7 Arthur Kreymer
export KRB5CCNAME="FILE:/tmp/krb5cc_sam_`whoami`_${$}" 
154 7 Arthur Kreymer
KEYTAB=/var/adm/krb5/`kcron -f`
155 7 Arthur Kreymer
KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d / | head -1`
156 7 Arthur Kreymer
kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
157 7 Arthur Kreymer
getcert -s
158 1 Katherine Lato
</pre>