Project

General

Profile

SHAREDJOBSUBSETUP

Setting up keytabs and proxies for jobsub in shared accounts

In certain cases we use shared accounts for production tasks:
  • production grid job submission
  • management of SAM metadata
  • data handling tasks

Because Fermigrid jobs and other tasks require Grid proxies tied to an individual,
we need to make such proxies available to the shared account.

An OSG Service Certificates can be used to make such a proxy,
and will be supported by jobsub_client in early April 2015.
See support documentation at
https://cdcvs.fnal.gov/redmine/projects/offline_production_operations_service/wiki/Configuring_shared_account_OSG_certificates

Meanwhile, the recommended technique is to copy a designated individual's keytab file into the shared account.
We place it in the path normally used by kcron, so that scripts can be written to run in both shared and individual accounts.
The method of generating and copying the keytab seems arcane and complicated.
There are reasons for each of the steps, so do everything described here !

In the following discussion we describe copying the kreymer keytab to minospro on gpsn01.fnal.gov .

Pick a user who will not likely be running grid jobs, as production use will modify the users' priorities.

New users of this method are urged to wait for OSG Service Cert proxy support in April 2015

We want to copy the kreymer kcron keytab file into the corresponding place in minospro.
  • The name of each account's kcron keytab file under /var/adm/krb5 is given by kcron -f .
  • File permissions prohibit the direct creation of the minospro keytab file.
  • Run kcroninit under the minospro account to create a keytab (give a fake password)
  • Copy the kreymer account keytab into the minospro keytab file
  1. kreymer opens two windows
  2. MINOSPRO
    • Create an invalid kcron keytab to be overwritten.
      kcroninit
      ...
      Are you on a secure channel?  (default = y): y
      What is your kerberos principal (default = kreymer@FNAL.GOV): 
      Enter the password for minospro@FNAL.GOV: junq
      

      [we do this so that the kcron keytab file for MINOSPRO gets created with the right permissions and we can update it later]
    • Check that the keytab was created
      ls -l /var/adm/krb5/`kcron -f`
      
  3. KREYMER
    • Verify that the existing kreymer keytab works with kcron
      kcron klist
      
      Ticket cache: FILE:/tmp/krb5cc_1060_IVnxb30674
      Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
      
      Valid starting     Expires            Service principal
      10/11/13 09:09:09  10/11/13 19:09:09  krbtgt/FNAL.GOV@FNAL.GOV
      ...
      

      The important bits here are that you
      • don't get an error message
      • the Default principal: line has your username/cron/hostname in it
      • the ticket expiration date is in the future
  4. KREYMER
    • Generate a command to be entered in the MINOSPRO window
      echo KEYUSE=/var/adm/krb5/`kcron -f`
      KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
      

      [Here we're locating your kcron keytab file]
  5. MINOSPRO Copy the kreymer keytab into of the invalid minospro keytab.
    KEYUSE=/var/adm/krb5/PbQYe9_Fl093H0sO6CRM1Q
    KEYSHARE=/var/adm/krb5/`kcron -f`
    scp kreymer@gpsn01.fnal.gov:${KEYUSE} ${KEYSHARE}
    

    [Here we locate MINOSPRO's keytab, and copy ours over it. We use scp rather than cp so we have permission to do so]
  6. MINOSPRO
    • Test the keytab, see that we have kreymer's cron principal
      KEYTAB=/var/adm/krb5/`kcron -f`
      KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d / | head -1`
      kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
      klist -f
      Ticket cache: FILE:/tmp/krb5cc_42411_ZneQA30945
      Default principal: kreymer/cron/gpsn01.fnal.gov@FNAL.GOV
      
      Valid starting     Expires            Service principal
      10/11/13 09:13:56  10/11/13 19:13:56  krbtgt/FNAL.GOV@FNAL.GOV
      ...
      

      Once again, we are checking that
      • we don't get an error message
      • the Default principal: line has your username/cron/hostname in it
      • the ticket expiration date is in the future
        [Here we're using kinit instead of kcron, because kcron will only look for and not ]
  7. MINOSPRO
    • Verify that we can get a production proxy for grid submission
      /scratch/grid/kproxy minos Production  # 
      /scratch/grid/kproxy -i
      ...
      lrwxrwxrwx 1 minospro gpcf   65 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
      -rw------- 1 minospro gpcf 7004 Oct 11 09:14 /scratch/minospro/grid/minospro.minos.production.proxy.2013101109
      lrwxrwxrwx 1 minospro gpcf   54 Oct 11 09:14 /scratch/minospro/grid/minospro.production.proxy -> /scratch/minospro/grid/minospro.minos.production.proxy
      ...
      /scratch/minospro/grid/minospro.minos.production.proxy
      attribute : /fermilab/minos/Role=Production/Capability=NULL
      Valid proxy expires in 43198 seconds (11 hours)
      Valid proxy expires at Fri Oct 11 21:14:42 CDT 2013
      
      
  8. MINOSPRO
    • use cron to automate the process --
      Run crontab -e to edit the crontab for your shared account, and add lines:
      1 1-23/2 * * * /scratch/grid/kproxy minos Production
      1 1-23/2 * * * /scratch/grid/kproxy minos Analysis
      

      (one for each role needed in the account)
      You should now have a proxy refreshed automatically.
  9. JOBSUB CLIENT
    When using the new jobsub_client in 2014, you do not run kproxy.
    Instead, you need to have an active kerberos ticket.
    Get this when needed within your cron scripts with
    export KRB5CCNAME="FILE:/tmp/krb5cc_jobsub_`whoami`_${$}" 
    KEYTAB=/var/adm/krb5/`kcron -f`
    KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d / | head -1`
    kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
    
    

Using the shared account for samweb access

Using the same procedure to get a shared ticket,
we can also access samweb from a shared account.

  1. Register the shared idenity with samweb, mapping to an appropriate account
    https://samweb.fnal.gov:8483/sam/minos/admin/users
        Added the account
        Added an appropriate Grid subject to that account, like
    /DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN=minos27.fnal.gov/CN=cron/CN=Arthur E. Kreymer/CN=UID:kreymer
    
  2. Get a ticket as with jobsub, with a unique ticket file
    export KRB5CCNAME="FILE:/tmp/krb5cc_sam_`whoami`_${$}" 
    KEYTAB=/var/adm/krb5/`kcron -f`
    KEYUSER=`klist -k ${KEYTAB} | grep FNAL.GOV | cut -c 5- | cut -f 1 -d / | head -1`
    kinit -5 -A  -kt ${KEYTAB} ${KEYUSER}/cron/`hostname`@FNAL.GOV
    getcert -s