Bug #23603

jobsub_server can't decode base64 strings with "-" in them as produced by the client

Added by Shreyas Bhat over 1 year ago. Updated about 1 year ago.

JobSub Server RPM
Target version:
Start date:
Due date:
% Done:


Estimated time:
(Total: 0.00 h)
First Occurred:
Occurs In:


In INC000001090158, Pengfei describes submitting a job using the following:

export RELEASE_NAME=v9_33_00&&export OUT_DIR=/pnfs/GM2/scratch/users/lwelty/hadd-tests/dingpf_endGame_20191114_103550&&export LIST_FILE_URL= NJOBS=1&&export HADD_METHOD=hadd&&export GRIDUSER=dingpf&&jobsub_submit -N 1 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC -e RELEASE_NAME -e OUT_DIR -e LIST_FILE_URL -e NJOBS -e HADD_METHOD -e GRIDUSER -G gm2 file:///gm2/app/users/dingpf/

The jobsub server returned an error saying that $RELEASE_NAME wasn't set. This isn't actually the issue.

What's happening is that jobsub_client is passing to the server an argument "--export-env", which is a base64-encoded string (using python's base64.urlsafe_b64encode method). This inserts "-" in certain spots. The options Pengfei passed in happened to yield an export string of:


On the jobsub server side, in we do something like "echo <b64_string> | base64 d" to decode the string. However, base64 on SL7 doesn't know how to handle the "" in the middle of the string above. Thus, the submission fails.

We need to change this decoding line so that it recognizes urlsafe base64 (

One possible workaround that seems to work on Pengfei's encoded string is:

echo <b64_string> | python -c "import base64, sys; [sys.stdout.write(base64.urlsafe_b64decode(line.strip())) for line in sys.stdin]" 

We'll need to think about this and see if there's a better way to go about this or if this will work. Hopefully, this will go out in the release due in mid-December.


Bug #23692: Review request [commit:5ab2faad865abaefa7e948b83797492ecfc93675: Changed base64-decoding lines to use python's base64 library instead of base64 command present in OS]ClosedDennis Box


#1 Updated by Shreyas Bhat over 1 year ago

  • Assignee set to Shreyas Bhat

#2 Updated by Shreyas Bhat over 1 year ago

  • % Done changed from 0 to 50

When we do the python3 porting, we'll need to use the line

echo $b64 | python3 -c "import base64, sys; [sys.stdout.write(str(base64.urlsafe_b64decode(line.strip()))) for line in sys.stdin]" 

Because python3's str.write() method expects a string.

#3 Updated by Shreyas Bhat over 1 year ago

Done and tested on fermicloud074.

I managed to test that the original line in this ticket worked, but not that the original case didn't.

Opening a ticket to Dennis to review.

#4 Updated by Shreyas Bhat over 1 year ago

  • % Done changed from 50 to 90
  • Status changed from New to Feedback

#5 Updated by Shreyas Bhat over 1 year ago

  • Start date changed from 11/14/2019 to 12/03/2019
  • Due date set to 12/03/2019

due to changes in a related task: #23692

#6 Updated by Shreyas Bhat over 1 year ago

Note: for python3, we'd have to use something like this (since sys.stdout.write expects a string and base64.urlsafe_b64decode gives a byte string:

echo <base64string> | python3 -c "import base64, sys; [sys.stdout.write(str(base64.urlsafe_b64decode(line.strip()))) for line in sys.stdin]" 

#7 Updated by Dennis Box about 1 year ago

Looks OK to merge

#8 Updated by Shreyas Bhat about 1 year ago

  • Status changed from Feedback to Resolved


Also available in: Atom PDF