Bug #23399

jobsub_client shouldn't care about date format when checking validity of kerberos tickets

Added by Shreyas Bhat 10 months ago. Updated 6 months ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)
First Occurred:
Occurs In:


Vito di Benedetto found out while testing jobsub client 1.2.10 on an SL7 node that running jobsub commands gave him the following error:

$ jobsub_q --debug --user vito
CLIENT_ARGS:  {'outFormat': None, 'uid': 'vito', 'constraint': None, 'acctRole': None, 'jobId': None, 'debug': True, 'acctGroup': None, 'better_analyze': False, 'jobsubServer': '', 'summary': False, 'help': False}
Using CA_DIR: /etc/grid-security/certificates
ltdict={'stime': '08/10/19 14:52:24', 'etime': '09/10/19 16:52:21'}

Cannot find credentials to use. Try the following:

- If you have an FNAL kerberized account, run 'kinit'.
- Otherwise, if you have an FNAL services account, run the following cigetcert command and which 
 will  prompt for your services password, then resubmit your job:
'cigetcert -s -o /tmp/x509up_u14191'
- Otherwise, follow the instructions at to obtain a services and/or kerberized account. 

What happened here was that jobsub will run (effectively) "klist -c `echo $KRB5CCNAME | cut -f 2 -d :`" to get the kerberos ticket cache info, and parse the first "Expires" line to see when the credentials expire. The date given by klist is something like "10/08/2019 16:40:30"

If $LC_TIME is set in the environment to some locale that's not American, the date format could be switched to a dd/mm/yyyy format rather than the American mm/dd/yyyy format (Vito's $LC_TIME was set to en_GB.utf8). This isn't an issue if $LC_TIME is unset either. Jobsub client looks for three date formats when evaluating kerberos credential validity, and none of these is not some form of dd/mm/yyyy. See

We need to make jobsub more intelligent. Rather than parsing the dates in jobsub, Dennis proposes that we let klist handle deciding whether or not a ticket is valid. If, for validity, we run "klist -c <cache> -s", a 0 return code tells us that the credential is valid, and anything else tells us it's not. We should change Krb5Ticket.expired() to simply run that klist command and act based on that return value.

We need jobsub


Bug #23499: Review request [commit:823beeb9552c2fdf0679affbebc55cee9fcfdc43: Catch CalledProcessError if raised when checking kerberos creds for validity]ClosedDennis Box


#1 Updated by Shreyas Bhat 9 months ago

  • % Done changed from 0 to 70
  • Status changed from New to Work in progress

Changes made and committed. Will reach out to Vito to test this before sending to Dennis for validation.

#2 Updated by Shreyas Bhat 9 months ago

  • % Done changed from 70 to 100

Vito tested successfully. We found that this is only an issue on SL7 nodes - SL6 klist doesn't respect LC_TIME.

Summary of testing results:
  1. With LC_TIME set to en_GB.utf8, kerb ticket parsing error raised in jobsub_client 1_2_10 when doing jobsub_q.
  2. Then, using this branch (git clone repo, checkout this branch), jobsub_q worked.
  3. kdestroy to get rid of kerb ticket, voms-proxy-destroy to get rid of proxy. Then we did "kinit -l 1s" to create a 1-second kerberos ticket (so we could test that the behavior was correct for an expired ticket). As expected, we got the error message in the original description of this ticket.

Ready for Dennis' validation.

#3 Updated by Shreyas Bhat 9 months ago

  • Start date changed from 10/08/2019 to 10/31/2019
  • Due date set to 10/31/2019

due to changes in a related task: #23499

#4 Updated by Shreyas Bhat 9 months ago

  • Target version deleted (v1.3.1)
  • Assignee deleted (Shreyas Bhat)
  • Status changed from Work in progress to Feedback
  • Category deleted (JobSub Client)

Opened review request for Dennis

#5 Updated by Dennis Box 9 months ago

  • Assignee set to Shreyas Bhat

Reviewed changes. OK to merge to 1.3.1

#6 Updated by Dennis Box 9 months ago

  • Status changed from Feedback to Accepted

#7 Updated by Dennis Box 7 months ago

  • Target version set to v1.3.1
  • Status changed from Accepted to Resolved

#8 Updated by Shreyas Bhat 6 months ago

  • Status changed from Resolved to Closed

This has been merged into the master branch and is in the jobsub_client 1.3.1 release.

Also available in: Atom PDF