Project

General

Profile

Using autorelease for more memory

For jobs which have fixed/no inputs: autorelease is available.

This is turned on on the Condor clusters, you can set a few extra lines on your job submission to enable it on your jobs, where if it gets held for memory, it can get auto-released with a larger memory request.

Note: I'm not sure if the Condor folks consider this tested enough for general use, so if you want to try it, contact them and let them know you're trying it.

you need to add to your jobsub line the following options:

      -l '+OriginalMemory=2000'
      -l '+GraceMemory=2000'
      -l '+MaxAllowedMemory=4000'
      -l 'request_memory=ifthenelse(isUndefined(MemoryUsage),OriginalMemory,MaxAllowedMemory)'
      -l 'periodic_release=(HoldReasonCode=?=34)&&(request_memory<MaxAllowedMemory)'

You can accomplish this with fife_launch by adding, in the "submit" section:

[submit]
...
lines_1 = '+OriginalMemory=2000'
lines_2 = '+GraceMemory=2000'
lines_3 = '+MaxAllowedMemory=4000'
lines_4 = 'request_memory=ifthenelse(isUndefined(MemoryUsage),OriginalMemory,MaxAllowedMemory)'
lines_5 = 'periodic_release=(HoldReasonCode=?=34)&&(request_memory<MaxAllowedMemory)'

You would of course make the initial and high memory values whatever you need, as opposed to 2000M and 4000M as above, and make GraceMemory the difference.