Bug #3103
Overly complicated secondary schedd config
0%
Description
Looking at the secondary schedd config created by both the ini installer and the RPMs,
it seems overly complicated.
We re-define the whole schedd world.
Why not use the environment, instead?
Like we do with the secondary collectors?
We know the needed values, since we used them in the Q&A mode.
Related issues
History
#1 Updated by Igor Sfiligoi over 8 years ago
We can replace
SCHEDDGLIDEINS5_ARGS = -local-name scheddglideins5 SCHEDD.SCHEDDGLIDEINS5.SCHEDD_NAME = schedd_glideins5 SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR = $(LOCAL_DIR)/$(SCHEDD.SCHEDDGLIDEINS5.SCHEDD_NAME) SCHEDD.SCHEDDGLIDEINS5.EXECUTE = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/execute SCHEDD.SCHEDDGLIDEINS5.LOCK = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/lock SCHEDD.SCHEDDGLIDEINS5.LOG = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/log SCHEDD.SCHEDDGLIDEINS5.PROCD_ADDRESS = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/procd_pipe SCHEDD.SCHEDDGLIDEINS5.SPOOL = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/spool SCHEDD.SCHEDDGLIDEINS5.JOB_QUEUE_LOG = $(SCHEDD.SCHEDDGLIDEINS5.SPOOL)/job_queue.log SCHEDD.SCHEDDGLIDEINS5.SCHEDD_ADDRESS_FILE = $(SCHEDD.SCHEDDGLIDEINS5.SPOOL)/.schedd_address SCHEDD.SCHEDDGLIDEINS5.SCHEDD_DAEMON_AD_FILE = $(SCHEDD.SCHEDDGLIDEINS5.SPOOL)/.schedd_classad SCHEDDGLIDEINS5_LOCAL_DIR_STRING = "$(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)" SCHEDD.SCHEDDGLIDEINS5.SCHEDD_EXPRS = LOCAL_DIR_STRING SCHEDDGLIDEINS5_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridManagerLog.$(SCHEDD.SCHEDDGLIDEINS5.SCHEDD_NAME).$(USERNAME)"
with just
SCHEDDGLIDEINS5_ENVIRONMENT = "_CONDOR_SCHEDD_NAME=schedd_glideins5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glideins5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glideins5/lock"
(copied pretty much verbatim from what we had in the scripts created by Q&A)
#2 Updated by Burt Holzman over 8 years ago
This is what John had to do to get it to work -- if your prescription works then we should use it (as long as it is wholly handled by the config file and condor manages the schedd startup, unlike the Q&A which launched them one at a time in a for loop)
#3 Updated by John Weigand over 8 years ago
Igor,
I assume you have this working in a test environment. Can you be more specific as to exactly what the complete set of attributes should be for a second schedd. I tried and failed based on what you had in this ticket. Likely me missing something. I would really like this to be simpler.
Thanks
John Weigand
#4 Updated by Igor Sfiligoi over 8 years ago
Ops... forgot about the shared daemon fix.
Looking at the latest Q&A installer, the right string to add should be
SCHEDDGLIDEINS5_ENVIRONMENT = \ "_CONDOR_SCHEDD_NAME=schedd_glideins5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glideins5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glideins5/lock _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(LOG)/shared_port_ad _CONDOR_DAEMON_SOCKET_DIR=$(LOG)/daemon_sock"
Sorry about that,
Igor
#5 Updated by Parag Mhashilkar over 8 years ago
I really don't see how one solution is more complicated than other. One defines a variable in each line while other defines all the variables in one line and that is converted to an extremely long environment line.
#6 Updated by Igor Sfiligoi over 8 years ago
- in the "Q&A mode", we set the base directories for the SCHEDD (+the preserve the shared port settings).
- in "current RPM mode". we set ALL the variables the SCHEDD may need to use (on the plus side, no need to tamper with shared port)
It is just much harder to maintain the "ALL options" variant;
we were hit already once, when Condor added an additional parameter to the schedd, and we did not define it... would not happen in the "Q&A mode".
Igor
#7 Updated by Parag Mhashilkar over 8 years ago
Igor Sfiligoi wrote:
The difference is WHAT we set;
- in the "Q&A mode", we set the base directories for the SCHEDD (+the preserve the shared port settings).
- in "current RPM mode". we set ALL the variables the SCHEDD may need to use (on the plus side, no need to tamper with shared port)
This is not about Q&A v/s RPM v/s ini installer issue. This is about how we should configure secondary schedd/collector and once we decide which way we want to go, installer(s) should be changed accordingly.
It is just much harder to maintain the "ALL options" variant;
we were hit already once, when Condor added an additional parameter to the schedd, and we did not define it... would not happen in the "Q&A mode".
If you are talking about lossing queue log, that was a nasty bug in condor and they fixed it. It's just a coincidence that in Q&A mode you didn't see it.
#8 Updated by Igor Sfiligoi over 8 years ago
BTW: I think I have found a way to simplify it further (and partially address Parag's concern).
We can define the shared port daemon settings only once, and then reuse them for all the schedds.
Unfortunately, concatenating strings does not work in the condor config, so we need to do a 2 step process.
SSP_SHPORT= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR) ... SCHEDDGLIDEIN1_ES = _CONDOR_SCHEDD_NAME=schedd_glidein1 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock $(SSP_SHPORT) SCHEDDGLIDEIN1_ENVIRONMENT = "$(STR_SCHEDDGLIDEIN1_ES)" ... SCHEDDGLIDEIN5_ES = _CONDOR_SCHEDD_NAME=schedd_glidein5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein5/lock $(SSP_SHPORT) SCHEDDGLIDEIN5_ENVIRONMENT = "$(STR_SCHEDDGLIDEIN5_ES)" ...
Now we really just define the straight minimum for each schedd ;)
#9 Updated by Igor Sfiligoi over 8 years ago
As for "Q&A vs RPM", I agree we should sync them.
Was just using those names to make it easier to refer to one vs the other way of writing configs.
#10 Updated by Igor Sfiligoi over 8 years ago
Sigh... the coffee is just starting to work.
Of course there is a way to inject a string into another string!
(but note that the SHPORT "string" is not really a string)
SSP_SHPORT= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR) ... SCHEDDGLIDEIN5_ENVIRONMENT = "_CONDOR_SCHEDD_NAME=schedd_glidein5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein5/lock $(SSP_SHPORT)" ...
#11 Updated by Parag Mhashilkar over 8 years ago
Just a data point after talking to Dan,
- Environment is inherited by children, whereas the other way is not.
- If you would ever want to reconfig any of these settings without restarting the daemon, this does not work the environment way.
I am perfectly fine either ways. I don't think it's going to affect the complexity by a big margin, its just necessary evil which we will have to deal with.
#12 Updated by John Weigand over 8 years ago
Igor,
Can you you expand on that last comment about your discussion with Dan about the environment?
Mainly the 2nd bullet. I have always had a hard time understanding when a condor_reconfig works and does not work.
Keep it simple especially for me.
Thanks
John Weigand
#13 Updated by Igor Sfiligoi over 8 years ago
Not much to say on the 2nd point from my part;
Dan just mentioned to me the same thing.
However, given that we only define directory/file locations via config/env, I don not expect the schedd would be happy we changed those without a restart anyhow.
PS: Point (1) is important though... I think the "current RPM/ini mode" does not properly configure the shadows... which the "Q&A mode i.e. env mode" does, due to child inheritance.
#14 Updated by John Weigand over 8 years ago
Igor,
Can you put in this ticket what you consider the "final" configuration entries of a single secondary schedd?
John Weigand
#15 Updated by Igor Sfiligoi over 8 years ago
Here is what I would put for 2 schedds, if we keep separate log directories (something I personally like a lot):
PRESERVE_SHPORT_EXPENV= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) \ _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR) # Secondary schedd schedd_glidein1 SCHEDDGLIDEIN1 = $(SCHEDD) SCHEDDGLIDEIN1_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein1 \ _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 \ _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock SCHEDDGLIDEIN1_ENVIRONMENT = "$(SCHEDDGLIDEIN1_EXPENV) $(PRESERVE_SHPORT_EXPENV)" SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN1 # Secondary schedd schedd_glidein2 SCHEDDGLIDEIN2 = $(SCHEDD) SCHEDDGLIDEIN2_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein2 \ _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein2 \ _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein2/lock SCHEDDGLIDEIN2_ENVIRONMENT = "$(SCHEDDGLIDEIN2_EXPENV) $(PRESERVE_SHPORT_EXPENV)" SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN2 DAEMON_LIST = $(DAEMON_LIST) $(SEC_SCHEDD_LIST) DC_DAEMON_LIST = + $(SEC_SCHEDD_LIST)
If instead you want all logs in the same dir, it becomes:
PRESERVE_SHPORT_EXPENV= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) \ _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR) # Secondary schedd schedd_glidein1 SCHEDDGLIDEIN1 = $(SCHEDD) SCHEDDGLIDEIN1_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein1 \ _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 \ _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock \ _CONDOR_SCHEDD_LOG=$(LOG)/SchedLog.schedd_glidein1 \ _CONDOR_SHADOW_LOG=$(LOG)/ShadowLog.schedd_glidein1 SCHEDDGLIDEIN1_ENVIRONMENT = "$(SCHEDDGLIDEIN1_EXPENV) $(PRESERVE_SHPORT_EXPENV)" SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN1 # Secondary schedd schedd_glidein2 SCHEDDGLIDEIN2 = $(SCHEDD) SCHEDDGLIDEIN2_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein2 \ _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein2 \ _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein2/lock \ _CONDOR_SCHEDD_LOG=$(LOG)/SchedLog.schedd_glidein2 \ _CONDOR_SHADOW_LOG=$(LOG)/ShadowLog.schedd_glidein2 SCHEDDGLIDEIN2_ENVIRONMENT = "$(SCHEDDGLIDEIN2_EXPENV) $(PRESERVE_SHPORT_EXPENV)" SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN2 DAEMON_LIST = $(DAEMON_LIST) $(SEC_SCHEDD_LIST) DC_DAEMON_LIST = + $(SEC_SCHEDD_LIST)
PS2: I expanded the list on multiple lines, since there were some comments against long lines.
PS2: This was obtained by running the script from #3101.
#16 Updated by Igor Sfiligoi over 8 years ago
Just thought of a slight improvement for the "all logs in the same dir" scenario:
PRESERVE_SHPORT_EXPENV= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) \ _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR) # Secondary schedd schedd_glidein1 SCHEDDGLIDEIN1 = $(SCHEDD) SCHEDDGLIDEIN1_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein1 \ _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 \ _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock \ _CONDOR_SCHEDD_LOG=$(SCHEDD_LOG).schedd_glidein1 \ _CONDOR_SHADOW_LOG=$(SHADOW_LOG).schedd_glidein1 SCHEDDGLIDEIN1_ENVIRONMENT = "$(SCHEDDGLIDEIN1_EXPENV) $(PRESERVE_SHPORT_EXPENV)" SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN1 # Secondary schedd schedd_glidein2 SCHEDDGLIDEIN2 = $(SCHEDD) SCHEDDGLIDEIN2_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein2 \ _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein2 \ _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein2/lock \ _CONDOR_SCHEDD_LOG=$(SCHEDD_LOG).schedd_glidein2 \ _CONDOR_SHADOW_LOG=$(SHADOW_LOG).schedd_glidein2 SCHEDDGLIDEIN2_ENVIRONMENT = "$(SCHEDDGLIDEIN2_EXPENV) $(PRESERVE_SHPORT_EXPENV)" SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN2 DAEMON_LIST = $(DAEMON_LIST) $(SEC_SCHEDD_LIST) DC_DAEMON_LIST = + $(SEC_SCHEDD_LIST)
#17 Updated by Burt Holzman about 8 years ago
- Target version set to v3_1
#18 Updated by Burt Holzman almost 8 years ago
- Assignee changed from Igor Sfiligoi to John Weigand
- Target version changed from v3_1 to v3_x
Since we won't ship the secondary schedds by default (but rely on the tool to build it), we just need to confirm that the tool builds the simpler config for 3.x.
#19 Updated by Parag Mhashilkar almost 6 years ago
- Assignee deleted (
John Weigand)