JobSub

Implement a retry policy in Condor to allow jobs that go held for excess memory usage to have their memory increased and released. Details copy/pasted from SNOW RITM0638375:

After discussing with HTCondor team here is what you want in the condor JDL. I haven't tested it so someone (Dennis or Marc or Shreyas) needs to test it.

# Supporting attributes that need to be added to JDL are shown by + sign
# OriginalMemory = <Value of --memory>
# GraceMemory = Increment request_memory by this amount if job was put on

+OriginalMemory = 2000
+GraceMemory = 2000
+MaxAllowedMemory = $(OriginalMemory) + $(GraceMemory)

request_memory = ifthenelse(isUndefined(MemoryUsage), OriginalMemory, MaxAllowedMemory)

periodic_release = (HoldReasonCode =?= 34) && (RequestMemory < MaxAllowedMemory)