Feature #19791: Implement a retry policy in Condor to allow jobs that go held for excess memory usage to have their memory increased and released - JobSub - Fermilab Redmine

Feature #19791

Implement a retry policy in Condor to allow jobs that go held for excess memory usage to have their memory increased and released

Added by Shreyas Bhat almost 3 years ago. Updated 3 months ago.

Status:

Closed

Priority:

Normal

Assignee:

Category:

JobSub Server RPM

Target version:

v1.3.4

Start date:

04/24/2018

Due date:

% Done:

Estimated time:

Stakeholders:

Duration:

Description

Implement a retry policy in Condor to allow jobs that go held for excess memory usage to have their memory increased and released. Details copy/pasted from SNOW RITM0638375:

After discussing with HTCondor team here is what you want in the condor JDL. I haven't tested it so someone (Dennis or Marc or Shreyas) needs to test it.

# Supporting attributes that need to be added to JDL are shown by + sign
# OriginalMemory = <Value of --memory>
# GraceMemory = Increment request_memory by this amount if job was put on

+OriginalMemory = 2000
+GraceMemory = 2000
+MaxAllowedMemory = $(OriginalMemory) + $(GraceMemory)

request_memory = ifthenelse(isUndefined(MemoryUsage), OriginalMemory, MaxAllowedMemory)

periodic_release = (HoldReasonCode =?= 34) && (RequestMemory < MaxAllowedMemory)