Project

General

Profile

SHAREDJOBSUBSETUP » History » Version 4

« Previous - Version 4/10 (diff) - Next » - Current version
Arthur Kreymer, 10/02/2014 12:42 PM
fixed typo


SHAREDJOBSUBSETUP

Setting up keytabs and proxies for jobsub in shared accounts

In certain cases we use shared accounts for production tasks:
  • production grid job submission
  • management of SAM metadata
  • data handling tasks

Because Fermigrid jobs and other tasks require Grid proxies tied to an individual,
we need to make that individual's proxies available to the shared account.

The recommended technique is to copy a designated individual's keytab file into the shared account.
We place it in the path normally used by kcron, so that scripts can be written to run in both shared and individual accounts.
The method of generating and copying the keytab seems arcane and complicated.
There are reasons for each of the steps, so do everything described here !

In the following discussion we describe copying the kreymer keytab to minospro on gpsn01.fnal.gov .

We want to simply copy the kreymer kcron keytab file into the corresponding place in minospro.
  • The name of each account's kcron keytab file under /var/adm/krb5 is given by kcron -f .
  • File permissions prohibit the direct creation of the minospro keytab file.
  • We run kcroninit under the minospro account to create a kcron keytab file.
  • This creates a new kreymer keytab, which invalidates the former kreymer keytab on this host.
  • So we run kcroninit in the kreymer account, which invalidates the keytab under minospro
  • So we copy the new kreymer account keytab over the now stale keytab in minospro.
  1. kreymer opens two windows
  2. MINOSPRO
    • Create a kcron keytab with kreymer's kcron identity.
      kcroninit
      

      [we do this so that the kcron keytab file for MINOSPRO gets created with the right permissions and we can update it later]
  3. KREYMER
    • Make a fresh kreymer kcron keytab
    • It is best to clear out the existing kcron keytab before making a new one.
      kcrondestroy
      kcroninit
      

      [We do this because running kcroninit for our kerberos principal in the other account invalidated this copy]
  4. KREYMER
    • Verify that the keytab works with kcron
      kcron klist
      
      Ticket cache: FILE:/tmp/krb5cc_1060_IVnxb30674
      Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
      
      Valid starting     Expires            Service principal
      10/11/13 09:09:09  10/11/13 19:09:09  krbtgt/FNAL.GOV@FNAL.GOV
      ...
      

      The important bits here are that you
      • don't get an error message
      • the Default principal: line has your username/cron/hostname in it
      • the ticket expiration date is in the future
  5. KREYMER
    • Generate a command to be entered in the MINOSPRO window
      echo KEYUSE=/var/adm/krb5/`kcron -f`
      KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
      

      [Here we're locating your kcron keytab file]
  6. MINOSPRO
    • enter the KEYUSE definition command echoed above
      KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
      
  7. MINOSPRO Copy the kreymer key on top of the now obsolete minospro keytab.
    KEYSHARE=/var/adm/krb5/`kcron -f`
    scp kreymer@gpsn01.fnal.gov:${KEYUSE} ${KEYSHARE}
    

    [Here we locate MINOSPRO's keytab, and copy ours over it. We use scp rather than cp so we have permission to do so]
  8. MINOSPRO
    • Test the keytab, see that we have kreymer's cron principal
      KEYTAB=/var/adm/krb5/`kcron -f`
      KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d /`
      kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
      klist -f
      Ticket cache: FILE:/tmp/krb5cc_42411_ZneQA30945
      Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
      
      Valid starting     Expires            Service principal
      10/11/13 09:13:56  10/11/13 19:13:56  krbtgt/FNAL.GOV@FNAL.GOV
      ...
      

      Once again, we are checking that
      • we don't get an error message
      • the Default principal: line has your username/cron/hostname in it
      • the ticket expiration date is in the future
        [Here we're using kinit instead of kcron, because kcron will only look for and not ]
  9. MINOSPRO
    • Verify that we can get a production proxy for grid submission
      /scratch/grid/kproxy minos Production  # 
      /scratch/grid/kproxy -i
      ...
      lrwxrwxrwx 1 minospro gpcf   65 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
      -rw------- 1 minospro gpcf 7004 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
      lrwxrwxrwx 1 minospro gpcf   54 Oct 11 09:14 /scratch/minospro/grid/minospro.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy
      ...
      /scratch/minospro/grid/minospro.minos.production.proxy
      attribute : /fermilab/minos/Role=Production/Capability=NULL
      Valid proxy expires in 43198 seconds (11 hours)
      Valid proxy expires at Fri Oct 11 21:14:42 CDT 2013
      
      
  10. MINOSPRO
    • use cron to automate the process --
      Run crontab -e to edit the crontab for your shared account, and add lines:
      1 1-23/2 * * * /scratch/grid/kproxy minos Production
      1 1-23/2 * * * /scratch/grid/kproxy minos Analysis
      

      (one for each role needed in the account)
      You should now have a proxy refreshed automatically.
  11. JOBSUB CLIENT
    When using the new jobsub_client in 2014, you do not run kproxy.
    Instead, you need to have an active kerberos ticket.
    Get this when needed within your cron scripts with
    export KRB5CCNAME="FILE:/tmp/krb5cc_jobsub_`whoami`_${$}" 
    KEYTAB=/var/adm/krb5/`kcron -f`
    KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d /`
    kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV