jobsub_lite Design Document

The basic design of jobsub_lite is straightforward. It will:

  • obtain credentials (using ifdhc, or use existing credentials)
  • parse command line arguments into a (Python) dictionary
  • optionally upload tarballs to the fast cvmfs distribution service
  • add other data to the dictionary from the environment, etc.
  • render (Jinja) templates with info from said dictionary to generate:
    • condor .cmd file(s)
    • job wrapper script(s)
    • condor dagman .dag files (in some cases)
  • use the python bindings to submit the above

There is also a simplified dagnabbit parser that reuses the same command line parser to generate .cmd and .sh files for each stage in the DAG.


This version of jobsub is expected to deal with SciTokens credentials; and will use the new ifdhc getToken call to fetch them, which in turn will call the htgettoken utility. Current thinking is that production accounts will have special production tokens pushed to them, and the utility will not have to get those tokens.

However, the initial implementation will try to pass either x509 credentials, or SciTokens, or both as the transition to SciTokens may be longer than we expect.

Command line parsing

This is done with the usual Python argument parser. Options supported are those from Dennis Box's summary of jobsub options actually used in the last 6 months.

Tarball uploads

This is currently implemented using Python's requests module for https access to the pubapi ; it is planned to improve it using the streaming mulipart uploader so we do not have to read the whole tarball into memory.

Tempates of .cmd .sh, and .dag files

The Jinja template code is used to generate the job submission files. For example, the template for the .cmd file for a simple submission is simple.cmd where a name (or expression) in doubled curly braces {{name}} is replaced when generating the output, and conditional geneartion is done with {%if expr%}...{%endif%}. (There are other Jinja features, but that is mainly what is used now in jobsub_lite). The majority of template replacement values are directly command line options or their defaults, which makes adding new features easy; you add an option (say --fred ) to the command line parser, and then add a suitable entry to the appropriate template(s) using that value ( {{fred}} or {%if fred %} xyzzy={{fred}} {%endif%} .


The new design is significantly smaller and easier to maintain than the current client/server code; as of this writing the prototype is under 700 lines of Python (although this could be expected to double as suitable error handling, etc. are added), and 1000 lines of template files, whereas the existing client / server code is over 5000 / 8000 lines, respectively; so we expect the released package to be about a factor of 10 smaller than the current implementation in lines of Python.