GlideinWMS

We can replace

SCHEDDGLIDEINS5_ARGS  = -local-name scheddglideins5
SCHEDD.SCHEDDGLIDEINS5.SCHEDD_NAME   = schedd_glideins5
SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR     = $(LOCAL_DIR)/$(SCHEDD.SCHEDDGLIDEINS5.SCHEDD_NAME)
SCHEDD.SCHEDDGLIDEINS5.EXECUTE       = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/execute
SCHEDD.SCHEDDGLIDEINS5.LOCK          = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/lock
SCHEDD.SCHEDDGLIDEINS5.LOG           = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/log
SCHEDD.SCHEDDGLIDEINS5.PROCD_ADDRESS = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/procd_pipe
SCHEDD.SCHEDDGLIDEINS5.SPOOL         = $(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)/spool
SCHEDD.SCHEDDGLIDEINS5.JOB_QUEUE_LOG   = $(SCHEDD.SCHEDDGLIDEINS5.SPOOL)/job_queue.log
SCHEDD.SCHEDDGLIDEINS5.SCHEDD_ADDRESS_FILE   = $(SCHEDD.SCHEDDGLIDEINS5.SPOOL)/.schedd_address
SCHEDD.SCHEDDGLIDEINS5.SCHEDD_DAEMON_AD_FILE = $(SCHEDD.SCHEDDGLIDEINS5.SPOOL)/.schedd_classad 
SCHEDDGLIDEINS5_LOCAL_DIR_STRING     = "$(SCHEDD.SCHEDDGLIDEINS5.LOCAL_DIR)" 
SCHEDD.SCHEDDGLIDEINS5.SCHEDD_EXPRS  = LOCAL_DIR_STRING

SCHEDDGLIDEINS5_ENVIRONMENT = "_CONDOR_GRIDMANAGER_LOG=$(LOG)/GridManagerLog.$(SCHEDD.SCHEDDGLIDEINS5.SCHEDD_NAME).$(USERNAME)" 

with just

SCHEDDGLIDEINS5_ENVIRONMENT = "_CONDOR_SCHEDD_NAME=schedd_glideins5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glideins5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glideins5/lock" 

This is what John had to do to get it to work -- if your prescription works then we should use it (as long as it is wholly handled by the config file and condor manages the schedd startup, unlike the Q&A which launched them one at a time in a for loop)

Igor,

I assume you have this working in a test environment. Can you be more specific as to exactly what the complete set of attributes should be for a second schedd. I tried and failed based on what you had in this ticket. Likely me missing something. I would really like this to be simpler.

Thanks
John Weigand

Ops... forgot about the shared daemon fix.

Looking at the latest Q&A installer, the right string to add should be

SCHEDDGLIDEINS5_ENVIRONMENT = \
"_CONDOR_SCHEDD_NAME=schedd_glideins5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glideins5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glideins5/lock _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(LOG)/shared_port_ad _CONDOR_DAEMON_SOCKET_DIR=$(LOG)/daemon_sock" 

Sorry about that,
Igor

I really don't see how one solution is more complicated than other. One defines a variable in each line while other defines all the variables in one line and that is converted to an extremely long environment line.

• in the "Q&A mode", we set the base directories for the SCHEDD (+the preserve the shared port settings).
• in "current RPM mode". we set ALL the variables the SCHEDD may need to use (on the plus side, no need to tamper with shared port)

It is just much harder to maintain the "ALL options" variant;
we were hit already once, when Condor added an additional parameter to the schedd, and we did not define it... would not happen in the "Q&A mode".

Igor

Igor Sfiligoi wrote:

The difference is WHAT we set;

• in the "Q&A mode", we set the base directories for the SCHEDD (+the preserve the shared port settings).
• in "current RPM mode". we set ALL the variables the SCHEDD may need to use (on the plus side, no need to tamper with shared port)

This is not about Q&A v/s RPM v/s ini installer issue. This is about how we should configure secondary schedd/collector and once we decide which way we want to go, installer(s) should be changed accordingly.

It is just much harder to maintain the "ALL options" variant;
we were hit already once, when Condor added an additional parameter to the schedd, and we did not define it... would not happen in the "Q&A mode".

If you are talking about lossing queue log, that was a nasty bug in condor and they fixed it. It's just a coincidence that in Q&A mode you didn't see it.

BTW: I think I have found a way to simplify it further (and partially address Parag's concern).

We can define the shared port daemon settings only once, and then reuse them for all the schedds.
Unfortunately, concatenating strings does not work in the condor config, so we need to do a 2 step process.

SSP_SHPORT= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR)
...
SCHEDDGLIDEIN1_ES = _CONDOR_SCHEDD_NAME=schedd_glidein1 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock $(SSP_SHPORT)
SCHEDDGLIDEIN1_ENVIRONMENT = "$(STR_SCHEDDGLIDEIN1_ES)" 
...
SCHEDDGLIDEIN5_ES = _CONDOR_SCHEDD_NAME=schedd_glidein5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein5/lock $(SSP_SHPORT)
SCHEDDGLIDEIN5_ENVIRONMENT = "$(STR_SCHEDDGLIDEIN5_ES)" 
...

Now we really just define the straight minimum for each schedd ;)

As for "Q&A vs RPM", I agree we should sync them.

Was just using those names to make it easier to refer to one vs the other way of writing configs.

Sigh... the coffee is just starting to work.
Of course there is a way to inject a string into another string!
(but note that the SHPORT "string" is not really a string)

SSP_SHPORT= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR)
...
SCHEDDGLIDEIN5_ENVIRONMENT = "_CONDOR_SCHEDD_NAME=schedd_glidein5 _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein5 _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein5/lock $(SSP_SHPORT)" 
...

Just a data point after talking to Dan,

• Environment is inherited by children, whereas the other way is not.
• If you would ever want to reconfig any of these settings without restarting the daemon, this does not work the environment way.

I am perfectly fine either ways. I don't think it's going to affect the complexity by a big margin, its just necessary evil which we will have to deal with.

Igor,

Can you you expand on that last comment about your discussion with Dan about the environment?
Mainly the 2nd bullet. I have always had a hard time understanding when a condor_reconfig works and does not work.
Keep it simple especially for me.

Thanks

John Weigand

Not much to say on the 2nd point from my part;
Dan just mentioned to me the same thing.

However, given that we only define directory/file locations via config/env, I don not expect the schedd would be happy we changed those without a restart anyhow.

PS: Point (1) is important though... I think the "current RPM/ini mode" does not properly configure the shadows... which the "Q&A mode i.e. env mode" does, due to child inheritance.

Igor,

Can you put in this ticket what you consider the "final" configuration entries of a single secondary schedd?

John Weigand

Here is what I would put for 2 schedds, if we keep separate log directories (something I personally like a lot):

PRESERVE_SHPORT_EXPENV= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) \
                        _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR)

# Secondary schedd schedd_glidein1
SCHEDDGLIDEIN1             = $(SCHEDD)
SCHEDDGLIDEIN1_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein1 \
                        _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 \
                        _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock
SCHEDDGLIDEIN1_ENVIRONMENT = "$(SCHEDDGLIDEIN1_EXPENV) $(PRESERVE_SHPORT_EXPENV)" 
SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN1

# Secondary schedd schedd_glidein2
SCHEDDGLIDEIN2             = $(SCHEDD)
SCHEDDGLIDEIN2_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein2 \
                        _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein2 \
                        _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein2/lock
SCHEDDGLIDEIN2_ENVIRONMENT = "$(SCHEDDGLIDEIN2_EXPENV) $(PRESERVE_SHPORT_EXPENV)" 
SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN2

DAEMON_LIST = $(DAEMON_LIST) $(SEC_SCHEDD_LIST)
DC_DAEMON_LIST = + $(SEC_SCHEDD_LIST)

If instead you want all logs in the same dir, it becomes:

PRESERVE_SHPORT_EXPENV= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) \
                        _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR)

# Secondary schedd schedd_glidein1
SCHEDDGLIDEIN1             = $(SCHEDD)
SCHEDDGLIDEIN1_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein1 \
                        _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 \
                        _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock \
                        _CONDOR_SCHEDD_LOG=$(LOG)/SchedLog.schedd_glidein1 \
                        _CONDOR_SHADOW_LOG=$(LOG)/ShadowLog.schedd_glidein1
SCHEDDGLIDEIN1_ENVIRONMENT = "$(SCHEDDGLIDEIN1_EXPENV) $(PRESERVE_SHPORT_EXPENV)" 
SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN1

# Secondary schedd schedd_glidein2
SCHEDDGLIDEIN2             = $(SCHEDD)
SCHEDDGLIDEIN2_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein2 \
                        _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein2 \
                        _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein2/lock \
                        _CONDOR_SCHEDD_LOG=$(LOG)/SchedLog.schedd_glidein2 \
                        _CONDOR_SHADOW_LOG=$(LOG)/ShadowLog.schedd_glidein2
SCHEDDGLIDEIN2_ENVIRONMENT = "$(SCHEDDGLIDEIN2_EXPENV) $(PRESERVE_SHPORT_EXPENV)" 
SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN2

DAEMON_LIST = $(DAEMON_LIST) $(SEC_SCHEDD_LIST)
DC_DAEMON_LIST = + $(SEC_SCHEDD_LIST)

PS2: I expanded the list on multiple lines, since there were some comments against long lines.
PS2: This was obtained by running the script from #3101.

Just thought of a slight improvement for the "all logs in the same dir" scenario:

PRESERVE_SHPORT_EXPENV= _CONDOR_SHARED_PORT_DAEMON_AD_FILE=$(SHARED_PORT_DAEMON_AD_FILE) \
                        _CONDOR_DAEMON_SOCKET_DIR=$(DAEMON_SOCKET_DIR)

# Secondary schedd schedd_glidein1
SCHEDDGLIDEIN1             = $(SCHEDD)
SCHEDDGLIDEIN1_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein1 \
                        _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein1 \
                        _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein1/lock \
                        _CONDOR_SCHEDD_LOG=$(SCHEDD_LOG).schedd_glidein1 \
                        _CONDOR_SHADOW_LOG=$(SHADOW_LOG).schedd_glidein1
SCHEDDGLIDEIN1_ENVIRONMENT = "$(SCHEDDGLIDEIN1_EXPENV) $(PRESERVE_SHPORT_EXPENV)" 
SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN1

# Secondary schedd schedd_glidein2
SCHEDDGLIDEIN2             = $(SCHEDD)
SCHEDDGLIDEIN2_EXPENV = _CONDOR_SCHEDD_NAME=schedd_glidein2 \
                        _CONDOR_LOCAL_DIR=$(LOCAL_DIR)/schedd_glidein2 \
                        _CONDOR_LOCK=$(LOCAL_DIR)/schedd_glidein2/lock \
                        _CONDOR_SCHEDD_LOG=$(SCHEDD_LOG).schedd_glidein2 \
                        _CONDOR_SHADOW_LOG=$(SHADOW_LOG).schedd_glidein2
SCHEDDGLIDEIN2_ENVIRONMENT = "$(SCHEDDGLIDEIN2_EXPENV) $(PRESERVE_SHPORT_EXPENV)" 
SEC_SCHEDD_LIST = $(SEC_SCHEDD_LIST) SCHEDDGLIDEIN2

DAEMON_LIST = $(DAEMON_LIST) $(SEC_SCHEDD_LIST)
DC_DAEMON_LIST = + $(SEC_SCHEDD_LIST)