- Table of contents
- Configure the Server
- Start the server
Configure the Server¶
Configuration Files¶
Git Repo for New or Changed Config Files¶
- git clone ssh://p-discompsupp@cdcvs.fnal.gov/cvs/projects/discompsupp-usdc_scripts
- cd jobsub
- config files to be installed as part of RIT will be tagged with RIT number
Config File Locations and Descriptions¶
- /opt/jobsub/server/conf/jobsub.ini
- main configuration file for jobsub server
- /etc/httpd/conf.d/jobsub_api.conf
- main configuration file for cherrypy engine that hosts jobsub server on port 8443
- /var/www/html/cigetcertopts.txt
- cigetcert options to jobsub_client needed for myproxy authentication
- /etc/httpd/conf.d/ssl.conf
- needed to serve cigetcertopts.txt on port 43
- /etc/lcmaps.db and /etc/lcmaps/lcmaps.db
- needed for lcmaps to authenticate correctly to gums
- /etc/sudoers
- must contain line: rexbatch ALL=(ALL) NOPASSWD:SETENV: /opt/jobsub/server/webapp/jobsub_priv *
- /etc/sysconfig/httpd
- must contain line: export OPENSSL_ALLOW_PROXY_CERTS=1
configure jobsub_api.conf settings¶
The relevant section of jobsub_api.conf:
WSGIDaemonProcess jobsub user=rexbatch group=fife processes=2 threads=25 python-path=/opt/jobsub/server/webapp:/opt/jobsub/lib/JobsubConfigParser:/opt/jobsub/lib/logger WSGIProcessGroup jobsub WSGIScriptAlias / /opt/jobsub/server/webapp/jobsub_api.py SetEnv JOBSUB_INI_FILE /opt/jobsub/server/conf/jobsub.ini SetEnv JOBSUB_UPS_LOCATION /fnal/ups/etc/setups.sh SetEnv JOBSUB_LOG_DIR /var/log/jobsub SetEnv JOBSUB_APP_NAME jobsub SetEnv JOBSUB_ENV_RUNNER /opt/jobsub/server/webapp/jobsub_env_runner.sh SetEnv JOBSUB_CREDENTIALS_DIR ~/.security SetEnv KADMIN_PASSWD_FILE ~/.security/kadmin_passwd SetEnv KCA_DN_PATTERN_LIST '^/DC=gov/DC=fnal/O=Fermilab/OU=People/CN.*/CN=UID:(.*$),^/DC=gov/DC=fnal/O=Fermilab/OU=Robots/CN.*/CN=cron/CN.*/CN=UID:(.*$)' SetEnv JOBSUB_SERVER_X509_CERT /etc/grid-security/jobsub/fifebatch-dev-hostcert.pem SetEnv JOBSUB_SERVER_X509_KEY /etc/grid-security/jobsub/fifebatch-dev-hostkey.pem
- If you are running the server as different uid:gid than rexbatch:fife edit the line 'WSGIDaemonProcess jobsub user=rexbatch group=fife ....' to the uid:gid you prefer.
- make sure JOBSUB_LOG_DIR exists and is writable by whoever is defined by WSGIDaemonProcess (grid:condor in this example). For the default install, that would mean the following 2 commands:
- mkdir -p /var/log/jobsub
- chown rexbatch:fife /var/log/jobsub
- consider enabling log rotation if this is a heavily used production machine.
configure jobsub.ini settings¶
configure the condor schedd¶
- edit the QUEUE_SUPER_USERS and QUEUE_SUPER_USER_MAY_IMPERSONATE value as below;
#whatever you need to talk to external collector, plus these two QUEUE_SUPER_USER values @ #nb if you change the server to run as some other uid than 'rexbatch' #you have to put that uid in the QUEUE_SUPER_USERS list instead of 'grid' @ QUEUE_SUPER_USERS = root,condor,rexbatch QUEUE_SUPER_USER_MAY_IMPERSONATE = .*
You will also need to configure the Glideinwms frontend to look for the jobs in this schedd
configure the cert and crl updates¶
- /usr/sbin/osg-ca-manage setupCA --location root --url osg
- /sbin/service osg-update-certs-cron start
- /sbin/chkconfig fetch-crl-boot on
- /sbin/chkconfig fetch-crl-cron on
enable proxy refresh script in a cron job¶
- /opt/jobsub/server/admin/krbrefresh.sh must be run from a cron job
- must be run from the account that runs the webserver i.e. 'grid' in this example, 'rexbatch' on fifebatch1.fnal.gov
- here is online help for krbrefresh.sh :
$ /opt/jobsub/server/admin/krbrefresh.sh ################################################################### file:krbrefresh.sh usage: krbrefresh.sh [ -h ] [--help] [--refresh-proxies ] [age_in_seconds] it must be run as user grid who has the ability to refresh user kerberos principals and voms-proxies in $JOBSUB_CREDENTIALS_DIR This script refreshes the kerberos proxies of any user in the queue that has a kerberos principal older than [age_in_seconds]. If no [age_in_seconds] argument is given, the default of 3600 seconds is used. This script logs its actions to file /opt/jobsub/server/log/admin.log ##################################################################
changes necessary for HA servers¶
- there are typically more than one HA servers sitting behind a DNS round-robin alias
- add a line for each server in the DNS round robin to allow the servers to authenticate condor commands to each other. In this example there are two servers, fife-jobsub-dev01 and fife-jobsub-dev02 sitting behind DNS alias fifebatch-dev.fnal.gov:
GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-jobsub-dev01.fnal.gov" schedd GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-jobsub-dev02.fnal.gov" schedd GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-gwmsfrontend-dev01.fnal.gov" frontend GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-uc-dev01.fnal.gov" collector GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fife-uc-dev02.fnal.gov" collector GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=frontend_pp/fifebatch.fnal.gov" frontend GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=cloudpilot/fifebatch.fnal.gov" cloudpilot GSI "/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=fifebatch-dev.fnal.gov" rexbatch GSI (.*) anonymous FS (.*) \1
Start the server¶
- service httpd start
- service condor start