Configure the Server¶

Configuration Files¶

Git Repo for New or Changed Config Files¶

git clone ssh://p-discompsupp@cdcvs.fnal.gov/cvs/projects/discompsupp-usdc_scripts
cd jobsub
config files to be installed as part of RIT will be tagged with RIT number

Config File Locations and Descriptions¶

/opt/jobsub/server/conf/jobsub.ini
- main configuration file for jobsub server
/etc/httpd/conf.d/jobsub_api.conf
- main configuration file for cherrypy engine that hosts jobsub server on port 8443
/var/www/html/cigetcertopts.txt
- cigetcert options to jobsub_client needed for myproxy authentication
/etc/httpd/conf.d/ssl.conf
- needed to serve cigetcertopts.txt on port 43
/etc/lcmaps.db and /etc/lcmaps/lcmaps.db
- needed for lcmaps to authenticate correctly to gums
/etc/sudoers
- must contain line: rexbatch ALL=(ALL) NOPASSWD:SETENV: /opt/jobsub/server/webapp/jobsub_priv *
/etc/sysconfig/httpd
- must contain line: export OPENSSL_ALLOW_PROXY_CERTS=1

configure jobsub_api.conf settings¶

The relevant section of jobsub_api.conf:


    WSGIDaemonProcess jobsub user=rexbatch group=fife processes=2 threads=25 python-path=/opt/jobsub/server/webapp:/opt/jobsub/lib/JobsubConfigParser:/opt/jobsub/lib/logger
    WSGIProcessGroup jobsub
    WSGIScriptAlias / /opt/jobsub/server/webapp/jobsub_api.py

    SetEnv JOBSUB_INI_FILE /opt/jobsub/server/conf/jobsub.ini
    SetEnv JOBSUB_UPS_LOCATION /fnal/ups/etc/setups.sh
    SetEnv JOBSUB_LOG_DIR /var/log/jobsub
    SetEnv JOBSUB_APP_NAME jobsub
    SetEnv JOBSUB_ENV_RUNNER /opt/jobsub/server/webapp/jobsub_env_runner.sh
    SetEnv JOBSUB_CREDENTIALS_DIR ~/.security
    SetEnv KADMIN_PASSWD_FILE ~/.security/kadmin_passwd
    SetEnv KCA_DN_PATTERN_LIST  '^/DC=gov/DC=fnal/O=Fermilab/OU=People/CN.*/CN=UID:(.*$),^/DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN.*/CN=cron/CN.*/CN=UID:(.*$)'
    SetEnv JOBSUB_SERVER_X509_CERT /etc/grid-security/jobsub/fifebatch-dev-hostcert.pem
    SetEnv JOBSUB_SERVER_X509_KEY  /etc/grid-security/jobsub/fifebatch-dev-hostkey.pem

If you are running the server as different uid:gid than rexbatch:fife edit the line 'WSGIDaemonProcess jobsub user=rexbatch group=fife ....' to the uid:gid you prefer.

make sure JOBSUB_LOG_DIR exists and is writable by whoever is defined by WSGIDaemonProcess (grid:condor in this example). For the default install, that would mean the following 2 commands:
- mkdir -p /var/log/jobsub
- chown rexbatch:fife /var/log/jobsub
consider enabling log rotation if this is a heavily used production machine.

configure jobsub.ini settings¶

configure the condor schedd¶

edit the QUEUE_SUPER_USERS and QUEUE_SUPER_USER_MAY_IMPERSONATE value as below;

#whatever you need to talk to external collector, plus these two QUEUE_SUPER_USER values
@
#nb if you change the server to run as some other uid than 'rexbatch' 
#you have to put that uid in the QUEUE_SUPER_USERS list instead of 'grid'
@
QUEUE_SUPER_USERS       = root,condor,rexbatch
QUEUE_SUPER_USER_MAY_IMPERSONATE = .*

You will also need to configure the Glideinwms frontend to look for the jobs in this schedd

configure the cert and crl updates¶

/usr/sbin/osg-ca-manage setupCA --location root --url osg
/sbin/service osg-update-certs-cron start
/sbin/chkconfig fetch-crl-boot on
/sbin/chkconfig fetch-crl-cron on

enable proxy refresh script in a cron job¶

/opt/jobsub/server/admin/krbrefresh.sh must be run from a cron job
must be run from the account that runs the webserver i.e. 'grid' in this example, 'rexbatch' on fifebatch1.fnal.gov
here is online help for krbrefresh.sh :

$ /opt/jobsub/server/admin/krbrefresh.sh
###################################################################
file:krbrefresh.sh
usage: krbrefresh.sh [ -h ] 
                     [--help] 
                     [--refresh-proxies ]  [age_in_seconds]

it must be run as user grid who has the ability to refresh user 
kerberos principals and voms-proxies in $JOBSUB_CREDENTIALS_DIR

This script refreshes the kerberos proxies of any user in the queue 
that has a kerberos principal older than [age_in_seconds].  If no
[age_in_seconds] argument is given, the default of 3600 seconds is used.

This script logs its actions to file /opt/jobsub/server/log/admin.log
##################################################################

changes necessary for HA servers¶

there are typically more than one HA servers sitting behind a DNS round-robin alias
add a line for each server in the DNS round robin to allow the servers to authenticate condor commands to each other. In this example there are two servers, fife-jobsub-dev01 and fife-jobsub-dev02 sitting behind DNS alias fifebatch-dev.fnal.gov:

GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-jobsub-dev01.fnal.gov" schedd
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-jobsub-dev02.fnal.gov" schedd
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-gwmsfrontend-dev01.fnal.gov" frontend
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-uc-dev01.fnal.gov" collector
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-uc-dev02.fnal.gov" collector
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=frontend_pp/fifebatch.fnal.gov" frontend
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=cloudpilot/fifebatch.fnal.gov" cloudpilot
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fifebatch-dev.fnal.gov" rexbatch
GSI (.*) anonymous
FS (.*) \1

Start the server¶

service httpd start
service condor start

Files

Project

General

Profile

JobSub

Wiki