Necessary Maintenance #22164
Upgrade jobsub code to use argparse instead of optparse
100%
Description
We want to use argparse to get jobsub ready to move to python 3. Some pointers for upgrading optparse to argparse are here:
https://docs.python.org/2.7/library/argparse.html#upgrading-optparse-code
With some playing around, it looks like opts, args = parser.parse_args() in optparse can become args, extras = parser.parse_known_args() pretty easily.
Related issues
History
#1 Updated by Shreyas Bhat about 2 years ago
- Blocks Bug #21031: jobsub_submit allows more than one --group argument added
#2 Updated by Shreyas Bhat about 2 years ago
- Status changed from New to Work in progress
#3 Updated by Shreyas Bhat about 2 years ago
jobsub_fetchlog, jobsub_submit, and jobsub_q done and tested.
#4 Updated by Shreyas Bhat about 2 years ago
jobsub_history done and tested.
#5 Updated by Shreyas Bhat almost 2 years ago
- % Done changed from 0 to 20
jobsub_hold and jobsub_probe_url done and tested.
#6 Updated by Shreyas Bhat almost 2 years ago
Rough outline of steps for each client module:
- imports (optparse, callbacks, version, etc)
- Usage line: %prog --> %(prog)s
- optparse --> argparse
- OptionParser --> ArgumentParser
- version_string: Comment out, add argument for Version_String
- opt_group --> client_options
- optparse.OptionGroup --> parser.add_argument_group; remove later parser.add_argument_group line
- add_option --> add_argument
- Change all types from strings to callables (like 'string' should become str)
- Change callbacks (action = callback --> action = actual callback class)
- Check that there's no "store_true" actions with metavars - argparse doesn't support that
- parser.parse_args --> parser.parse_known_args
- logSupport.dPrint.....: options --> vars(options)
#7 Updated by Shreyas Bhat almost 2 years ago
The rest of the client code is updated, but my dev client/server isn't working as I expected. I assume it's something to do with the incommon/curl issues we'd seen. I'll first rebase the work here off of current master, since I'd need to anyway, and then troubleshoot my dev server.
#8 Updated by Shreyas Bhat almost 2 years ago
- % Done changed from 20 to 40
There was a bug in the jobsub_submit_dag code that I fixed. Ready to move onto the server.
#9 Updated by Shreyas Bhat almost 2 years ago
- % Done changed from 40 to 60
Moved the groupsettings code and the server/tools code over to argparse. I need to test this code on my dev server.
I'll also need to move over the packaging/ code over. After that, I think I'm done.
#10 Updated by Shreyas Bhat over 1 year ago
Working example:
[12/Aug/2019:16:22:27] [139633514936064:job.py:doPOST] jobsub_args : /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162227.789056_322/probe sleep 600
[12/Aug/2019:16:22:27] [139633514936064:jobsub.py:execute_job_submit_wrapper] jobsub command: ['/opt/jobsub/server/webapp/jobsub_env_runner.sh', u'', u'/fife/local/scratch/uploads/nova/sbhat/2019-08-12_162227.789056_322/probe', u'sleep', u'600']
[12/Aug/2019:16:22:27] [139633514936064:condor_commands.py:schedd_list] condor_status -schedd -af name -constraint 'stringListMember(name,"fermicloud074.fnal.gov")'
[12/Aug/2019:16:22:27] [139633514936064:jobsub.py:run_cmd_as_user] /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162227.789056_322/probe sleep 600
[12/Aug/2019:16:22:28] [139633514936064:jobsub.py:execute_job_submit_wrapper] 131.225.155.98 sbhat@FNAL.GOV sbhat ? jobsub_submit --group nova probe sleep 600 JobsubJobId of first job: 367.0@fermicloud074.fnal.gov
[12/Aug/2019:16:22:28] [139633514936064:job.py:doPOST] JobsubJobId of first job: 367.0@fermicloud074.fnal.gov
Failed Example:
[12/Aug/2019:16:20:48] [139933148784384:jobsub.py:execute_job_submit_wrapper] jobsub command: ['/opt/jobsub/server/webapp/jobsub_env_runner.sh', u'', u'/fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe', u'sleep', u'600']
[12/Aug/2019:16:20:48] [139933148784384:condor_commands.py:schedd_list] condor_status -schedd -af name -constraint 'stringListMember(name,"fermicloud074.fnal.gov")'
[12/Aug/2019:16:20:48] [139933148784384:jobsub.py:run_cmd_as_user] /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600
[12/Aug/2019:16:20:48] [139933148784384:jobsub.py:run_cmd_as_user] Error running as user sbhat using command /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600:
STDOUT:
STDERR:
Exception:Command 'sudo -u sbhat -E /opt/jobsub/server/webapp/jobsub_priv runCommand /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600' returned non-zero exit status 1:
EXITCODE:1
STDOUT:
STDERR:Error running command: /opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600
ERROR: 'runCommand' failed with exception: Command '/opt/jobsub/server/webapp/jobsub_env_runner.sh '' /fife/local/scratch/uploads/nova/sbhat/2019-08-12_162048.008939_4431/probe sleep 600' returned non-zero exit status 1:
EXITCODE:1
STDOUT:_resource_callback() got an unexpected keyword argument 'dest'
What's going on here? That'll be my next thing to look at.
Right now, I've left fermicloud074 with the "working" version. Will need to stage the "test version" to debug.
#11 Updated by Shreyas Bhat over 1 year ago
Fixed this bug. My testing is all done - going to do a rebase and then send to Dennis for review.
#12 Updated by Shreyas Bhat over 1 year ago
- Status changed from Work in progress to Feedback
Rebase done. Opening a review request to Dennis.
#13 Updated by Shreyas Bhat over 1 year ago
- Related to Feature #23116: Review request [commit:6828275f7235c14c1220e65be943ef2f83978693: Fixed _resource_callback code to accept kwargs] added
#14 Updated by Shreyas Bhat over 1 year ago
- Blocks Feature #7313: Support regex in jobids for certain client commands added
#15 Updated by Shreyas Bhat over 1 year ago
- Blocks Feature #22099: jobsub constraint commands should respect --group even if --constraint doesn't include it: INCLUDING GLOBAL SUPERUSERS added
#16 Updated by Dennis Box over 1 year ago
- Status changed from Feedback to Accepted
- Tracker changed from Feature to Necessary Maintenance
Reviewed and tested code, passes the integration tests.
Merged to master as I need to branch off of this for my python2->3 work
#17 Updated by Shreyas Bhat over 1 year ago
- % Done changed from 60 to 100
- Status changed from Accepted to Resolved
Deployed in 1.3
#18 Updated by Shreyas Bhat over 1 year ago
- Status changed from Resolved to Closed