JobSub

Liang Li from gm2 noted that in the gpvm $TMPDIRs (usually /tmp), they're limited to 2 GB of space. Thus, if they try to create larger tarballs using the jobsub_client tardir:// feature, they can fill up tmp before the tarball is actually created.

This is because in client/jobsubClient.py, we create a temp file using tempfile.mktemp(), put the contents of the intended tarball in there, compress the tarball using gzip -n, and then move that file into place. Can we skip making the tempfile, and instead write directly to the final tarfile compressed using something like:

tar = tarfile.open(filename, 'w:gz')

Here is the original message from Liang:

        I do have one comment, tardir:// option seems to first tar the
designated directory and then compress it, all of which is done at $TMPDIR
(normally "/tmp"), this is actually a problem for gm2 VM --- for some
reason, /tmp space is limited to merely 2GB for all VMs (Adam and I are
starting another discussion about that). This has caused problems when /tmp
is filled up. Apparently /tmp can be easily filled up when tardir:// option
is used (as I explained above, a tar ball is *first* created and then
*compressed*). Of course, one can simply relocate $TMPDIR to circumvent
that. But I just thought that it might be more convenient (and probably more
efficient) for tardir:// option to act like a "tar cfz" command (which
creates tar ball and compresses it at the same time).