Project

General

Profile

Feature #11395

Ability to peacefully shutdown glidein if the worker node is marked for draining

Added by Parag Mhashilkar over 4 years ago. Updated about 4 years ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Start date:
01/11/2016
Due date:
% Done:

100%

Estimated time:
Stakeholders:

CMS

Duration:

Description

In WLCG, there is a file on the worker node that tells if the WN will be drained. Glidein can look at this file to initiate its draining. Also, investigate and support similar functionality for OSG

History

#1 Updated by Parag Mhashilkar over 4 years ago

Greg Thain, mailed my following recipe we can use.

Parag:

I heard from Miron that glideinWMS has a requirement to be able to have worker nodes stop accepting new work if the base system tells it to, perhaps by creating a file.  Todd asked me to mock up an example of how to do this with startd cron, and it is pretty straightforward.

First, we'll create a startd attribute called "StartJobs", which will get advertised as part of the startd, by adding it to the config file.  Then we'll use STARTD_ATTRS to have the startd advertise it, so we can query it externally with condor_status. Finally, we'll make this knob part of the start expression, so if it goes to false, START will go false, and the startd will not accept any new jobs.  (It will, however, run any existing jobs to completion):

StartJobs = true
STARTD_ATTRS = $(STARTD_ATTRS) StartJobs
START = StartJobs && $(START)

Then, in the same startd config file, we'll add a startd cron job to query the existing of a file, and set the value of StartdJobs appropriately,  These lines should do the trick.

STARTD_CRON_JOBLIST = $(STARTD_CRON_JOBLIST) DRAINER
STARTD_CRON_DRAINER_EXECUTABLE = /tmp/drainer.sh
STARTD_CRON_DRAINER_PERIOD = 30

Finally, the drainer.sh will just check for the existance of a file, and advertise StartJobs approriately:

#!/bin/sh

if [ -f /tmp/drainme ]
then
echo StartJobs = false
else
echo StartJobs = true
fi

exit 0

Let us know if you have any questions about this,

_greg

#3 Updated by Parag Mhashilkar over 4 years ago

One minor change I would like to propose for Greg's solution is to invert the semantics of the StartJobs variable to be more clear. For example, use something like, GLIDEIN_SiteWMS_WN_Draining

#4 Updated by Marco Mascheroni over 4 years ago

I have tried this out and it seems to work (forget what I said at the meeting, I fixed it)

The only thing that I find strange is that the startd process did not exit once the START expression evaluated to false, it hanged around for ~10 minutes. Still, no jobs were matched and both the startd and the pilot exited in less than 10 minutes, so I guess everything is ok. I put some logs at the end of the message if necessary [1], ignore them if you think everything is ok.

About the code, I think there are many way to do the same thing and I cannot really tell which is the correct one. I picked the simplest one (for me) and I changed the condor_config.dedicated_starter.include file like this:

[mmascher@lxplus0074 glideinwms]$ git diff
diff --git a/creation/web_base/condor_config.dedicated_starter.include b/creation/web_base/condor_config.dedicated_starter.include
index 5f22e0a..a53d8f1 100644
--- a/creation/web_base/condor_config.dedicated_starter.include
+++ b/creation/web_base/condor_config.dedicated_starter.include
@@ -15,11 +15,17 @@ HAS_MONITOR_VM=True
 IS_MONITOR_VM = False
 USES_MONITOR_STARTD = True

-STARTD_ATTRS = $(STARTD_ATTRS), IS_MONITOR_VM, HAS_MONITOR_VM, USES_MONITOR_STARTD, Monitoring_Name
+GLIDEIN_SiteWMS_WN_Draining = False
+
+STARTD_ATTRS = $(STARTD_ATTRS), IS_MONITOR_VM, HAS_MONITOR_VM, USES_MONITOR_STARTD, Monitoring_Name, GLIDEIN_SiteWMS_WN_Draining
+
+STARTD_CRON_JOBLIST = $(STARTD_CRON_JOBLIST) DRAINER
+STARTD_CRON_DRAINER_EXECUTABLE = /tmp/drainer.sh
+STARTD_CRON_DRAINER_PERIOD = 30s

 GLIDEIN_REQUIREMENTS = ((GLIDEIN_ToRetire =?= UNDEFINED) || (CurrentTime < GLIDEIN_ToRetire))

-START = ($(GLIDEIN_START_CONDITION)) && ($(GLIDEIN_REQUIREMENTS))
+START = ($(GLIDEIN_START_CONDITION)) && ($(GLIDEIN_REQUIREMENTS)) && (GLIDEIN_SiteWMS_WN_Draining =?= False)
 WANT_HOLD = ($(GLIDEIN_HOLD_CONDITION))
 PREEMPT = (($(GLIDEIN_HOLD_CONDITION)) || ($(GLIDEIN_PREEMPT_CONDITION)))

and then I put the /tmp/drainer.sh script in tmp. I can really use some indication about how ths should be done (Parag?). Like, should I put the drainer.sh as a snippet inside the condor_startup.sh script? Should I modify another condor_config file?

Thanks,

Marco Mascheroni

[1]
condor_q and condor_status at the submission node (user pool)

[mmascher@fermicloud344 sleepjob]$ condor_q
-- Schedd: fermicloud344.fnal.gov : <131.225.154.234:9615?...
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
  88.0   frontend        1/27 11:26   0+00:00:00 I  0   0.0  sleep.sh 60
  89.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  90.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  91.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  92.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  93.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  94.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  95.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  96.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  97.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60
  98.0   frontend        1/27 11:36   0+00:00:00 I  0   0.0  sleep.sh 60

11 jobs; 0 completed, 0 removed, 11 idle, 0 running, 0 held, 0 suspended

[mmascher@fermicloud344 sleepjob]$ condor_status
Name               OpSys      Arch   State     Activity LoadAv Mem   ActvtyTime

glidein_582249_553 LINUX      X86_64 Unclaimed Idle      0.040 1858  0+00:00:04
                     Machines Owner Claimed Unclaimed Matched Preempting

        X86_64/LINUX        1     0       0         1       0          0

               Total        1     0       0         1       0          0

Negotiator log on the submission node.

==> /var/log/condor/NegotiatorLog <==
01/27/16 11:50:15 ---------- Started Negotiation Cycle ----------
01/27/16 11:50:15 Phase 1:  Obtaining ads from collector ...
01/27/16 11:50:15 Not considering preemption, therefore constraining idle machines with ifThenElse(State == "Claimed","Name State Activity StartdIpAddr AccountingGroup Owner RemoteUser Requirements SlotWeight ConcurrencyLimits","")
01/27/16 11:50:15   Getting startd private ads ...
01/27/16 11:50:15   Getting Scheduler, Submitter and Machine ads ...
01/27/16 11:50:15   Sorting 3 ads ...
01/27/16 11:50:15 Got ads: 3 public and 1 private
01/27/16 11:50:15 Public ads include 1 submitter, 1 startd
01/27/16 11:50:15 Phase 2:  Performing accounting ...
01/27/16 11:50:15 Phase 3:  Sorting submitter ads by priority ...
01/27/16 11:50:15 Phase 4.1:  Negotiating with schedds ...
01/27/16 11:50:15   Negotiating with frontend@fermicloud344.fnal.gov at <131.225.154.234:9615?addrs=131.225.154.234-9615&noUDP&sock=773241_e733_6>
01/27/16 11:50:15 0 seconds so far for this submitter
01/27/16 11:50:15 0 seconds so far for this schedd
01/27/16 11:50:15     Got NO_MORE_JOBS;  schedd has no more requests
01/27/16 11:50:15     Request 00088.00000:  (request count 1 of 11)
01/27/16 11:50:15       Rejected 88.0 frontend@fermicloud344.fnal.gov <131.225.154.234:9615?addrs=131.225.154.234-9615&noUDP&sock=773241_e733_6>: no match found
01/27/16 11:50:15     Got NO_MORE_JOBS;  schedd has no more requests
01/27/16 11:50:15  negotiateWithGroup resources used scheddAds length 0
01/27/16 11:50:15 ---------- Finished Negotiation Cycle ----------

Startd log after the WN was set to draining.

[mmascher@fermicloud092 glidein-debug]$ tail /home/mmascher/glidein-debug/osg_wn_tmp/glide_wSyGSH/log/StartdLog
01/27/16 11:48:18 (pid:586212) Machine Requirements check failed!
01/27/16 11:48:18 (pid:586212) Called deactivate_claim()
01/27/16 11:48:18 (pid:586212) State change: received RELEASE_CLAIM command
01/27/16 11:48:18 (pid:586212) Changing state and activity: Claimed/Idle -> Preempting/Vacating
01/27/16 11:48:18 (pid:586212) State change: No preempting claim, returning to owner
01/27/16 11:48:18 (pid:586212) Changing state and activity: Preempting/Vacating -> Owner/Idle
01/27/16 11:48:18 (pid:586212) State change: IS_OWNER is false
01/27/16 11:48:18 (pid:586212) Changing state: Owner -> Unclaimed

#5 Updated by Marco Mascheroni over 4 years ago

  • Assignee changed from Marco Mascheroni to Marco Mambelli

#6 Updated by Parag Mhashilkar over 4 years ago

  • Status changed from New to Feedback

#7 Updated by Marco Mambelli over 4 years ago

  • Assignee changed from Marco Mambelli to Marco Mascheroni

Feedback sent via email
change back when ready.

#8 Updated by Parag Mhashilkar over 4 years ago

  • Target version changed from v3_2_13 to v3_2_14

#9 Updated by Marco Mascheroni about 4 years ago

  • Assignee changed from Marco Mascheroni to Marco Mambelli

#10 Updated by Marco Mambelli about 4 years ago

  • Assignee changed from Marco Mambelli to Marco Mascheroni

ready to merge

#11 Updated by Marco Mascheroni about 4 years ago

  • % Done changed from 0 to 100

#12 Updated by Marco Mascheroni about 4 years ago

  • Status changed from Feedback to Resolved

#13 Updated by Parag Mhashilkar about 4 years ago

  • Status changed from Resolved to Assigned

Marco, I looked at the v3/11395_2 and looks like there are changes in the branch that have not been merged back. So now I am confused about the status of the ticket. Can you please verify/merge and update the status accordingly?

#14 Updated by Marco Mascheroni about 4 years ago

  • Status changed from Assigned to Resolved

Done

#15 Updated by Parag Mhashilkar about 4 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF