Project

General

Profile

History

Wiki ยป

Configure the Server

Configuration Files

Git Repo for New or Changed Config Files

  • git clone ssh:///cvs/projects/discompsupp-usdc_scripts
  • cd jobsub
  • config files to be installed as part of RIT will be tagged with RIT number

Config File Locations and Descriptions

  • /opt/jobsub/server/conf/jobsub.ini
    • main configuration file for jobsub server
  • /etc/httpd/conf.d/jobsub_api.conf
    • main configuration file for cherrypy engine that hosts jobsub server on port 8443
  • /var/www/html/cigetcertopts.txt
    • cigetcert options to jobsub_client needed for myproxy authentication
  • /etc/httpd/conf.d/ssl.conf
    • needed to serve cigetcertopts.txt on port 43
  • /etc/lcmaps.db and /etc/lcmaps/lcmaps.db
    • needed for lcmaps to authenticate correctly to gums
  • /etc/sudoers
    • must contain line: rexbatch ALL=(ALL) NOPASSWD:SETENV: /opt/jobsub/server/webapp/jobsub_priv *
  • /etc/sysconfig/httpd
    • must contain line: export OPENSSL_ALLOW_PROXY_CERTS=1

configure jobsub_api.conf settings

The relevant section of jobsub_api.conf:


    WSGIDaemonProcess jobsub user=rexbatch group=fife processes=2 threads=25 python-path=/opt/jobsub/server/webapp:/opt/jobsub/lib/JobsubConfigParser:/opt/jobsub/lib/logger
    WSGIProcessGroup jobsub
    WSGIScriptAlias / /opt/jobsub/server/webapp/jobsub_api.py

    SetEnv JOBSUB_INI_FILE /opt/jobsub/server/conf/jobsub.ini
    SetEnv JOBSUB_UPS_LOCATION /fnal/ups/etc/setups.sh
    SetEnv JOBSUB_LOG_DIR /var/log/jobsub
    SetEnv JOBSUB_APP_NAME jobsub
    SetEnv JOBSUB_ENV_RUNNER /opt/jobsub/server/webapp/jobsub_env_runner.sh
    SetEnv JOBSUB_CREDENTIALS_DIR ~/.security
    SetEnv KADMIN_PASSWD_FILE ~/.security/kadmin_passwd
    SetEnv KCA_DN_PATTERN_LIST  '^/DC=gov/DC=fnal/O=Fermilab/OU=People/CN.*/CN=UID:(.*$),^/DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN.*/CN=cron/CN.*/CN=UID:(.*$)'
    SetEnv JOBSUB_SERVER_X509_CERT /etc/grid-security/jobsub/fifebatch-dev-hostcert.pem
    SetEnv JOBSUB_SERVER_X509_KEY  /etc/grid-security/jobsub/fifebatch-dev-hostkey.pem
  • If you are running the server as different uid:gid than rexbatch:fife edit the line 'WSGIDaemonProcess jobsub user=rexbatch group=fife ....' to the uid:gid you prefer.
  • make sure JOBSUB_LOG_DIR exists and is writable by whoever is defined by WSGIDaemonProcess (grid:condor in this example). For the default install, that would mean the following 2 commands:
    • mkdir -p /var/log/jobsub
    • chown rexbatch:fife /var/log/jobsub
  • consider enabling log rotation if this is a heavily used production machine.

configure jobsub.ini settings

configure the condor schedd

  • edit the QUEUE_SUPER_USERS and QUEUE_SUPER_USER_MAY_IMPERSONATE value as below;
    #whatever you need to talk to external collector, plus these two QUEUE_SUPER_USER values
@
#nb if you change the server to run as some other uid than 'rexbatch' 
#you have to put that uid in the QUEUE_SUPER_USERS list instead of 'grid'
@
QUEUE_SUPER_USERS       = root,condor,rexbatch
QUEUE_SUPER_USER_MAY_IMPERSONATE = .*

You will also need to configure the Glideinwms frontend to look for the jobs in this schedd

configure the cert and crl updates

  • /usr/sbin/osg-ca-manage setupCA --location root --url osg
  • /sbin/service osg-update-certs-cron start
  • /sbin/chkconfig fetch-crl-boot on
  • /sbin/chkconfig fetch-crl-cron on

enable proxy refresh script in a cron job

  • /opt/jobsub/server/admin/krbrefresh.sh must be run from a cron job
  • must be run from the account that runs the webserver i.e. 'grid' in this example, 'rexbatch' on fifebatch1.fnal.gov
  • here is online help for krbrefresh.sh :
$ /opt/jobsub/server/admin/krbrefresh.sh
###################################################################
file:krbrefresh.sh
usage: krbrefresh.sh [ -h ] 
                     [--help] 
                     [--refresh-proxies ]  [age_in_seconds]

it must be run as user grid who has the ability to refresh user 
kerberos principals and voms-proxies in $JOBSUB_CREDENTIALS_DIR

This script refreshes the kerberos proxies of any user in the queue 
that has a kerberos principal older than [age_in_seconds].  If no
[age_in_seconds] argument is given, the default of 3600 seconds is used.

This script logs its actions to file /opt/jobsub/server/log/admin.log
##################################################################

changes necessary for HA servers

  • there are typically more than one HA servers sitting behind a DNS round-robin alias
  • add a line for each server in the DNS round robin to allow the servers to authenticate condor commands to each other. In this example there are two servers, fife-jobsub-dev01 and fife-jobsub-dev02 sitting behind DNS alias fifebatch-dev.fnal.gov:
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-jobsub-dev01.fnal.gov" schedd
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-jobsub-dev02.fnal.gov" schedd
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-gwmsfrontend-dev01.fnal.gov" frontend
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-uc-dev01.fnal.gov" collector
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-uc-dev02.fnal.gov" collector
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=frontend_pp/fifebatch.fnal.gov" frontend
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=cloudpilot/fifebatch.fnal.gov" cloudpilot
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fifebatch-dev.fnal.gov" rexbatch
GSI (.*) anonymous
FS (.*) \1

Start the server

  • service httpd start
  • service condor start