Project

General

Profile

Bug #23399

jobsub_client shouldn't care about date format when checking validity of kerberos tickets

Added by Shreyas Bhat about 1 month ago. Updated 14 days ago.

Status:
Feedback
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
10/31/2019
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
First Occurred:
Occurs In:
Stakeholders:
Duration:

Description

Vito di Benedetto found out while testing jobsub client 1.2.10 on an SL7 node that running jobsub commands gave him the following error:

$ jobsub_q --debug --user vito
CLIENT_ARGS:  {'outFormat': None, 'uid': 'vito', 'constraint': None, 'acctRole': None, 'jobId': None, 'debug': True, 'acctGroup': None, 'better_analyze': False, 'jobsubServer': 'https://fifebatch.fnal.gov:8443', 'summary': False, 'help': False}
Using CA_DIR: /etc/grid-security/certificates
ltdict={'stime': '08/10/19 14:52:24', 'etime': '09/10/19 16:52:21'}
None

Cannot find credentials to use. Try the following:

- If you have an FNAL kerberized account, run 'kinit'.
- Otherwise, if you have an FNAL services account, run the following cigetcert command and which 
 will  prompt for your services password, then resubmit your job:
'cigetcert -s fifebatch.fnal.gov -o /tmp/x509up_u14191'
- Otherwise, follow the instructions at https://fermi.service-now.com/kb_view_customer.do?sysparm_article=KB0010798 to obtain a services and/or kerberized account. 

What happened here was that jobsub will run (effectively) "klist -c `echo $KRB5CCNAME | cut -f 2 -d :`" to get the kerberos ticket cache info, and parse the first "Expires" line to see when the credentials expire. The date given by klist is something like "10/08/2019 16:40:30"

If $LC_TIME is set in the environment to some locale that's not American, the date format could be switched to a dd/mm/yyyy format rather than the American mm/dd/yyyy format (Vito's $LC_TIME was set to en_GB.utf8). This isn't an issue if $LC_TIME is unset either. Jobsub client looks for three date formats when evaluating kerberos credential validity, and none of these is not some form of dd/mm/yyyy. See https://cdcvs.fnal.gov/redmine/projects/jobsub/repository/revisions/v1.2.10/entry/client/jobsubClientCredentials.py#L198

We need to make jobsub more intelligent. Rather than parsing the dates in jobsub, Dennis proposes that we let klist handle deciding whether or not a ticket is valid. If, for validity, we run "klist -c <cache> -s", a 0 return code tells us that the credential is valid, and anything else tells us it's not. We should change Krb5Ticket.expired() to simply run that klist command and act based on that return value.

We need jobsub


Subtasks

Bug #23499: Review request [commit:823beeb9552c2fdf0679affbebc55cee9fcfdc43: Catch CalledProcessError if raised when checking kerberos creds for validity]NewDennis Box

History

#1 Updated by Shreyas Bhat 15 days ago

  • % Done changed from 0 to 70
  • Status changed from New to Work in progress

Changes made and committed. Will reach out to Vito to test this before sending to Dennis for validation.

#2 Updated by Shreyas Bhat 14 days ago

  • % Done changed from 70 to 100

Vito tested successfully. We found that this is only an issue on SL7 nodes - SL6 klist doesn't respect LC_TIME.

Summary of testing results:
  1. With LC_TIME set to en_GB.utf8, kerb ticket parsing error raised in jobsub_client 1_2_10 when doing jobsub_q.
  2. Then, using this branch (git clone repo, checkout this branch), jobsub_q worked.
  3. kdestroy to get rid of kerb ticket, voms-proxy-destroy to get rid of proxy. Then we did "kinit -l 1s" to create a 1-second kerberos ticket (so we could test that the behavior was correct for an expired ticket). As expected, we got the error message in the original description of this ticket.

Ready for Dennis' validation.

#3 Updated by Shreyas Bhat 14 days ago

  • Start date changed from 10/08/2019 to 10/31/2019
  • Due date set to 10/31/2019

due to changes in a related task: #23499

#4 Updated by Shreyas Bhat 14 days ago

  • Target version deleted (v1.3.1)
  • Assignee deleted (Shreyas Bhat)
  • Status changed from Work in progress to Feedback
  • Category deleted (JobSub Client)

Opened review request for Dennis



Also available in: Atom PDF