Project

General

Profile

Necessary Maintenance #22164

Upgrade jobsub code to use argparse instead of optparse

Added by Shreyas Bhat 8 months ago. Updated about 2 months ago.

Status:
Accepted
Priority:
Normal
Assignee:
Shreyas Bhat
Category:
-
Target version:
v1.3
Start date:
03/19/2019
Due date:
% Done:

60%

Estimated time:
Stakeholders:
Duration:

Description

We want to use argparse to get jobsub ready to move to python 3. Some pointers for upgrading optparse to argparse are here:

https://docs.python.org/2.7/library/argparse.html#upgrading-optparse-code

With some playing around, it looks like opts, args = parser.parse_args() in optparse can become args, extras = parser.parse_known_args() pretty easily.

Related issues

Related to JobSub - Feature #23116: Review request [commit:6828275f7235c14c1220e65be943ef2f83978693: Fixed _resource_callback code to accept kwargs]New08/13/2019

Blocks JobSub - Bug #21031: jobsub_submit allows more than one --group argument Work in progress11/29/2018

Blocks JobSub - Feature #7313: Support regex in jobids for certain client commandsNew11/12/2014

Blocks JobSub - Feature #22099: jobsub constraint commands should respect --group even if --constraint doesn't include it: INCLUDING GLOBAL SUPERUSERSNew03/08/2019

History

#1 Updated by Shreyas Bhat 8 months ago

  • Blocks Bug #21031: jobsub_submit allows more than one --group argument added

#2 Updated by Shreyas Bhat 7 months ago

  • Status changed from New to Work in progress

#3 Updated by Shreyas Bhat 7 months ago

jobsub_fetchlog, jobsub_submit, and jobsub_q done and tested.

#4 Updated by Shreyas Bhat 7 months ago

jobsub_history done and tested.

#5 Updated by Shreyas Bhat 7 months ago

  • % Done changed from 0 to 20

jobsub_hold and jobsub_probe_url done and tested.

#6 Updated by Shreyas Bhat 7 months ago

Rough outline of steps for each client module:

  1. imports (optparse, callbacks, version, etc)
  2. Usage line: %prog --> %(prog)s
  3. optparse --> argparse
  4. OptionParser --> ArgumentParser
  5. version_string: Comment out, add argument for Version_String
  6. opt_group --> client_options
  7. optparse.OptionGroup --> parser.add_argument_group; remove later parser.add_argument_group line
  8. add_option --> add_argument
  9. Change all types from strings to callables (like 'string' should become str)
  10. Change callbacks (action = callback --> action = actual callback class)
  11. Check that there's no "store_true" actions with metavars - argparse doesn't support that
  12. parser.parse_args --> parser.parse_known_args
  13. logSupport.dPrint.....: options --> vars(options)

#7 Updated by Shreyas Bhat 6 months ago

The rest of the client code is updated, but my dev client/server isn't working as I expected. I assume it's something to do with the incommon/curl issues we'd seen. I'll first rebase the work here off of current master, since I'd need to anyway, and then troubleshoot my dev server.

#8 Updated by Shreyas Bhat 6 months ago

  • % Done changed from 20 to 40

There was a bug in the jobsub_submit_dag code that I fixed. Ready to move onto the server.

#9 Updated by Shreyas Bhat 5 months ago

  • % Done changed from 40 to 60

Moved the groupsettings code and the server/tools code over to argparse. I need to test this code on my dev server.

I'll also need to move over the packaging/ code over. After that, I think I'm done.

#10 Updated by Shreyas Bhat 3 months ago

Working example:

[12/Aug/2019:16:22:27] [139633514936064:job.py:doPOST] jobsub_args : /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162227.789056_322/probe sleep 600
[12/Aug/2019:16:22:27] [139633514936064:jobsub.py:execute_job_submit_wrapper] jobsub command: ['/opt/jobsub/server/webapp/jobsub_env_runner.sh', u'', u'/fife/local/scratch/uploads/nova/sbhat/2019-08-12_162227.789056_322/probe', u'sleep', u'600']
[12/Aug/2019:16:22:27] [139633514936064:condor_commands.py:schedd_list] condor_status -schedd -af name -constraint 'stringListMember(name,"fermicloud074.fnal.gov")'
[12/Aug/2019:16:22:27] [139633514936064:jobsub.py:run_cmd_as_user] /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162227.789056_322/probe sleep 600
[12/Aug/2019:16:22:28] [139633514936064:jobsub.py:execute_job_submit_wrapper] 131.225.155.98 sbhat ? jobsub_submit --group nova probe sleep 600 JobsubJobId of first job:

[12/Aug/2019:16:22:28] [139633514936064:job.py:doPOST] JobsubJobId of first job:

Failed Example:

[12/Aug/2019:16:20:48] [139933148784384:jobsub.py:execute_job_submit_wrapper] jobsub command: ['/opt/jobsub/server/webapp/jobsub_env_runner.sh', u'', u'/fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe', u'sleep', u'600']
[12/Aug/2019:16:20:48] [139933148784384:condor_commands.py:schedd_list] condor_status -schedd -af name -constraint 'stringListMember(name,"fermicloud074.fnal.gov")'
[12/Aug/2019:16:20:48] [139933148784384:jobsub.py:run_cmd_as_user] /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600
[12/Aug/2019:16:20:48] [139933148784384:jobsub.py:run_cmd_as_user] Error running as user sbhat using command /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600:
STDOUT:
STDERR:
Exception:Command 'sudo -u sbhat -E /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600' returned non-zero exit status 1:
EXITCODE:1
STDOUT:
STDERR:Error running command: /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600
ERROR: 'runCommand' failed with exception: Command '/opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600' returned non-zero exit status 1:
EXITCODE:1
STDOUT:_resource_callback() got an unexpected keyword argument 'dest'

What's going on here? That'll be my next thing to look at.

Right now, I've left fermicloud074 with the "working" version. Will need to stage the "test version" to debug.

#11 Updated by Shreyas Bhat 3 months ago

Fixed this bug. My testing is all done - going to do a rebase and then send to Dennis for review.

#12 Updated by Shreyas Bhat 3 months ago

  • Status changed from Work in progress to Feedback

Rebase done. Opening a review request to Dennis.

#13 Updated by Shreyas Bhat 3 months ago

  • Related to Feature #23116: Review request [commit:6828275f7235c14c1220e65be943ef2f83978693: Fixed _resource_callback code to accept kwargs] added

#14 Updated by Shreyas Bhat 3 months ago

  • Blocks Feature #7313: Support regex in jobids for certain client commands added

#15 Updated by Shreyas Bhat 3 months ago

  • Blocks Feature #22099: jobsub constraint commands should respect --group even if --constraint doesn't include it: INCLUDING GLOBAL SUPERUSERS added

#16 Updated by Dennis Box about 2 months ago

  • Status changed from Feedback to Accepted
  • Tracker changed from Feature to Necessary Maintenance

Reviewed and tested code, passes the integration tests.
Merged to master as I need to branch off of this for my python2->3 work

Also available in: Atom PDF