JobSub

In INC000001090158, Pengfei describes submitting a job using the following:

export RELEASE_NAME=v9_33_00&&export OUT_DIR=/pnfs/GM2/scratch/users/lwelty/hadd-tests/dingpf_endGame_20191114_103550&&export LIST_FILE_URL=http://home.fnal.gov/~dingpf/20191114_103550.list&&export NJOBS=1&&export HADD_METHOD=hadd&&export GRIDUSER=dingpf&&jobsub_submit -N 1 --resource-provides=usage_model=DEDICATED,OPPORTUNISTIC -e RELEASE_NAME -e OUT_DIR -e LIST_FILE_URL -e NJOBS -e HADD_METHOD -e GRIDUSER -G gm2 file:///gm2/app/users/dingpf/hadd_grid_script.sh

The jobsub server returned an error saying that $RELEASE_NAME wasn't set. This isn't actually the issue.

What's happening is that jobsub_client is passing to the server an argument "--export-env", which is a base64-encoded string (using python's base64.urlsafe_b64encode method). This inserts "-" in certain spots. The options Pengfei passed in happened to yield an export string of:

ZXhwb3J0IEdSSURVU0VSPWRpbmdwZjtleHBvcnQgSEFERF9NRVRIT0Q9aGFkZDtleHBvcnQgTkpPQlM9MTtleHBvcnQgTElTVF9GSUxFX1VSTD1odHRwOi8vaG9tZS5mbmFsLmdvdi9-ZGluZ3BmLzIwMTkxMTE0XzEwMzU1MC5saXN0O2V4cG9ydCBPVVRfRElSPS9wbmZzL0dNMi9zY3JhdGNoL3VzZXJzL2x3ZWx0eS9oYWRkLXRlc3RzL2RpbmdwZl9lbmRHYW1lXzIwMTkxMTE0XzEwMzU1MDtleHBvcnQgUkVMRUFTRV9OQU1FPXY5XzMzXzAwOw==

On the jobsub server side, in jobsub_env_runner.py we do something like "echo <b64_string> | base64 d" to decode the string. However, base64 on SL7 doesn't know how to handle the "" in the middle of the string above. Thus, the submission fails.

We need to change this decoding line so that it recognizes urlsafe base64 (https://en.wikipedia.org/wiki/Base64#Variants_summary_table).

One possible workaround that seems to work on Pengfei's encoded string is:

echo <b64_string> | python -c "import base64, sys; [sys.stdout.write(base64.urlsafe_b64decode(line.strip())) for line in sys.stdin]" 

We'll need to think about this and see if there's a better way to go about this or if this will work. Hopefully, this will go out in the release due in mid-December.