SAM Web Test Environment

These notes assume you have your own SLF6 VM setup with FermiCloud. These notes are for SAM developers. Keep in mind that samdev is the experiment which has its own database.

Get In VM & Do Initial Setting Up


Since we don't have root access to our machines in production, it's important that you don't do this as root, but as yourself.
ssh fermicloud060

source /fnal/ups/etc/

which ups

Become root.

We now need to install GIT.
yum install git

Now exit to return back to your user.

Get The SAM Web Code

We grab the postgres branch from the repository. Eventually this branch will be the master branch.
git clone ssh:///cvs/projects/sam-web
Now get into the sam-web directory.

cd sam-web/

Take Care Of Dependencies

ups depend sam_web -r $PWD -m ups/sam_web.table

The command just ran will produce lots of error messages. This is how we know what dependencies to install.
We now want to become the products user.
ksu products

To see where the products are living you can use df.
df ~products

You may need to source ups again.
. /fnal/ups/etc/

setup upd

Time to find out what ups products are available to install.
upd list python -aK+

upd install sam v9_0_5

You will get messages asking you to manually run commands so be sure to do them.
ups declare -f NULL -q "" -g current sam_ns_ior v7_2_3 -z /fnal/ups/db

ups declare -f NULL -q "" -g current sam_config v7_1_7 -z /fnal/ups/db

Let's now install CherryPy.
upd install cherrypy v3_8_0

Install the postgres client.
upd install postgres_client v9_3_5_p2_7

Install the database driver.
upd install psycopg2 v2_5_p2_7 -j

Install the ORM.
upd install sqlalchemy v1_0_12

Install the template language.
upd install mako v1_0_0

Install the Protocol Buffer.
upd install protobuf v2_5_0 -q cxx11:gcc4.8:python2.7

Now stop being the products user.

Time to be root so we can install some packages.

Install openssl compatibility library.
yum list openssl*

yum install openssl098e.x86_64

Now stop being the root user.

Protocol Buffer Submodule

At this point there is nothing in the sam_protobuf directory. We need to add our submodule to it.

git submodule update --init

If the command just ran fails, it's because you don't have access to the repository.
Now if you do an ls on the sam_protobuf directory you will now see a couple files there.


git clone ssh:///cvs/projects/sam-web-plugins

Setup SAM Web

Get into the sam-web directory.

cd sam-web/

setup sam_web -r $PWD -m ups/sam_web.table

You will get a bunch of errors, but this is OK. Run the command again.
setup sam_web -r $PWD -m ups/sam_web.table

You will get a message "No default SAM configuration exists at this time. Again, this is OK.

The following is the equivalent, so you can pick whatever you are comfortable with:

setup -. sam_web


Create a file in the sam-web directory called samdev.conf. It should like below.


server.socket_host = ''
server.socket_port = 20020
#log.screen = True

samwebapi = "/sam/samdev" 

tools.proxy.on = True


experiment = 'samdev'
new_dimensions = True
#new_subrun will override "experiment='minerva'" 

default_project_url = '/api/running/projects/'

admin_users = ['illingwo','swhite', 'mgheith']


role = 'samdev'
database = 'samdev'
host = ''
port = '5433'

Start SAM Web Server

Do the section above if you haven't done so (Setup SAM Web). We now start the SAM Web Server with the above configuration file.

samweb_server samdev.conf

If it started properly you can now do the following and get a response back.
curl http://localhost:20020/sam/samdev/api

Install Nose & Run Tests

All the tests live in the test directory. Before you can run tests you need to install Nose first.

ksu products

. /fnal/ups/etc/

setup upd

upd list nose -aK+

ls -l /fnal/ups/prd/

upd list nose -aK+

upd install nose v1_3_0 -G-c

Now exit as products user.
To run the tests do the following below.
nosetests test/

Or if you want to run in verbose mode you can do this.
nosetests -v test/

Or if you do not want to see the test results, but the logs in action do.
nosetests --nologcapture test/

Or if you want to see identifier information and run all tests do.
nosetests --with-id -v test/

If you run into problems, please check samdev.conf that lives in the test directory. You may need to update the database details.

Create Your Own Setup Script

For getting things up quickly just source a bash script that does everything for you. We create it now in our sam-web directory.


. /fnal/ups/etc/
setup sam_web -r $PWD -m ups/sam_web.table

export SAM_DB_SERVER_NAME=SAMDbServer.samdev_invalid:SAMDbServer

Then when you want to get things going just do the following.
samweb_server samdev.conf

Or you can do the following while in the sam-web directory:

. /fnal/ups/etc/
setup -. sam_web
samweb_server samdev.conf

Apache Server Install With Needed Modules

My node already had httpd installed already, but you will want to run these commands and install the necessary packages.


yum list httpd

yum install httpd

yum list *http*

yum list *mod* OR yum list mod_*

yum install mod_ssl

Now stop being root.

SAM Web Apache Configuration

The configuration file is /etc/httpd/conf.d/sam_web_servers.conf and currently looks like the following.

#For the proxy worker sharing to work properly we have to declare the proxying outside the virtual hosts
# This means that all virtual hosts will expose all these downstream servers

ProxyTimeout 3600
RewriteEngine On

# dev servers
ProxyPass /sam/samdev http://localhost:20020/sam/samdev ttl=60 retry=0
ProxyPass /sam/minerva/dev  http://localhost:20003/sam/minerva/dev retry=0

# prd servers
# To try an avoid timeout errors from the backends, the max connections per server should
# be no more than the max number of cherrypy threads servicing the connections. This parameter is
# per apache worker, so for cherrypy with 10 threads and 2 workers the limit should be 5
# If I understand the documentation correctly, since the ProxyPass definitions are the same
# the normal and SSL VirtualServers will share worker pools
#ProxyPass /sam/minerva    http://localhost:20000/sam/minerva max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/nova       http://localhost:20001/sam/nova    max=15 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/dzero      http://localhost:20007/sam/dzero   max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/cdf        http://localhost:20009/sam/cdf     max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/uboone     http://localhost:20011/sam/uboone  max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/gm2        http://localhost:20013/sam/gm2     max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPassMatch ^/sam/(darkside|ds50)(.*) http://localhost:20015/sam/darkside$2 max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/lbne       http://localhost:20017/sam/lbne    max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/minos      http://localhost:20019/sam/minos   max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/coupp      http://localhost:20021/sam/coupp   max=5 connectiontimeout=1800 ttl=5 retry=0
#ProxyPass /sam/mu2e       http://localhost:20024/sam/mu2e    max=5 connectiontimeout=1800 ttl=5 retry=0

#ProxyPass /sam_web_registry http://localhost:30000/sam_web_registry max=5 connectiontimeout=1800 ttl=5 retry=0

SSL Configuration

The configuration file is /etc/httpd/conf.d/port8483_ssl.conf and currently looks like the following.

# This is the Apache server configuration file providing SSL support.
# It contains the configuration directives to instruct the server how to
# serve pages over an https connection. For detailing information about these 
# directives see <URL:>
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  

# Proxy definitions are here
LoadModule ssl_module modules/

# When we also provide SSL we have to listen to the 
# the HTTPS port in addition.
Listen 8483

##  SSL Global Context
##  All SSL configuration in this context applies both to
##  the main server and all SSL-enabled virtual hosts.

#   Some MIME-types for downloading Certificates and CRLs
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

#   Pass Phrase Dialog:
#   Configure the pass phrase gathering process.
#   The filtering dialog program (`builtin' is a internal
#   terminal dialog) has to provide the pass phrase on stdout.
SSLPassPhraseDialog  builtin

#   Inter-Process Session Cache:
#   Configure the SSL Session Cache: First the mechanism 
#   to use and second the expiring timeout (in seconds).
#SSLSessionCache        dc:UNIX:/var/cache/mod_ssl/distcache
SSLSessionCache         shmcb:/var/cache/mod_ssl/scache(512000)
SSLSessionCacheTimeout  300

#   Semaphore:
#   Configure the path to the mutual exclusion semaphore the
#   SSL engine uses internally for inter-process synchronization. 
SSLMutex default

#   Pseudo Random Number Generator (PRNG):
#   Configure one or more sources to seed the PRNG of the 
#   SSL library. The seed data should be of good random quality.
#   WARNING! On some platforms /dev/random blocks if not enough entropy
#   is available. This means you then cannot use the /dev/random device
#   because it would lead to very long connection times (as long as
#   it requires to make more entropy available). But usually those
#   platforms additionally provide a /dev/urandom device which doesn't
#   block. So, if available, use this one instead. Read the mod_ssl User
#   Manual for more details.
SSLRandomSeed startup file:/dev/urandom  256
SSLRandomSeed connect builtin
#SSLRandomSeed startup file:/dev/random  512
#SSLRandomSeed connect file:/dev/random  512
#SSLRandomSeed connect file:/dev/urandom 512

# Use "SSLCryptoDevice" to enable any supported hardware
# accelerators. Use "openssl engine -v" to list supported
# engine names.  NOTE: If you enable an accelerator and the
# server does not start, consult the error logs and ensure
# your accelerator is functioning properly. 
SSLCryptoDevice builtin
#SSLCryptoDevice ubsec

## SSL Virtual Host Context

<VirtualHost _default_:8483>

# General setup for the virtual host, inherited from global configuration
#DocumentRoot "/var/www/html" 

# Use separate log files for the SSL virtual host; note that LogLevel
# is not inherited from httpd.conf.
#ErrorLog logs/ssl_error_log
#TransferLog logs/ssl_access_log
#ErrorLog  "|/usr/bin/tee -a /home/sam/httpd/logs/ssl_error_log  | /usr/bin/logger -thttpd -plocal6.err" 
CustomLog "|/usr/bin/tee -a /home/sam/httpd/logs/ssl_access_log | /usr/bin/logger -thttpd -plocal6.notice" combined
LogLevel warn
#LogLevel debug

#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   SSL Protocol support:
# List the enable protocol levels with which clients will be able to
# connect.  Disable SSLv2 access by default:
SSLProtocol all -SSLv2 -SSLv3

#   SSL Cipher Suite:
# List the ciphers that the client is permitted to negotiate.
# See the mod_ssl documentation for a complete list.

#   Server Certificate:
# Point SSLCertificateFile at a PEM encoded certificate.  If
# the certificate is encrypted, then you will be prompted for a
# pass phrase.  Note that a kill -HUP will prompt again.  A new
# certificate can be generated using the genkey(1) command.
SSLCertificateFile /home/sam/httpd/certs/samweb_cert.pem

#   Server Private Key:
#   If the key is not combined with the certificate, use this
#   directive to point at the key file.  Keep in mind that if
#   you've both a RSA and a DSA private key you can configure
#   both in parallel (to also allow the use of DSA ciphers, etc.)
SSLCertificateKeyFile /home/sam/httpd/certs/samweb_key.pem

#   Server Certificate Chain:
#   Point SSLCertificateChainFile at a file containing the
#   concatenation of PEM encoded CA certificates which form the
#   certificate chain for the server certificate. Alternatively
#   the referenced file can be the same as SSLCertificateFile
#   when the CA certificates are directly appended to the server
#   certificate for convinience.
#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

#   Certificate Authority (CA):
#   Set the CA certificate verification path where to find CA
#   certificates for client authentication or alternatively one
#   huge file containing all of them (file must be PEM encoded)
#SSLCACertificateFile /etc/pki/tls/certs/ca-bundle.crt

#   Client Authentication (Type):
#   Client certificate verification type and depth.  Types are
#   none, optional, require and optional_no_ca.  Depth is a
#   number which specifies how deeply to verify the certificate
#   issuer chain before deciding the certificate is not valid.
#SSLVerifyClient require
#SSLVerifyDepth  10

#   Access Control:
#   With SSLRequire you can do per-directory access control based
#   on arbitrary complex boolean expressions containing server
#   variable checks and other lookup directives.  The syntax is a
#   mixture between C and Perl.  See the mod_ssl documentation
#   for more details.
#<Location />
#SSLRequire (    %{SSL_CIPHER} !~ m/^(EXP|NULL)/ \
#            and %{SSL_CLIENT_S_DN_O} eq "Snake Oil, Ltd." \
#            and %{SSL_CLIENT_S_DN_OU} in {"Staff", "CA", "Dev"} \
#            and %{TIME_WDAY} >= 1 and %{TIME_WDAY} <= 5 \
#            and %{TIME_HOUR} >= 8 and %{TIME_HOUR} <= 20       ) \
#           or %{REMOTE_ADDR} =~ m/^192\.76\.162\.[0-9]+$/

#   SSL Engine Options:
#   Set various options for the SSL engine.
#   o FakeBasicAuth:
#     Translate the client X.509 into a Basic Authorisation.  This means that
#     the standard Auth/DBMAuth methods can be used for access control.  The
#     user name is the `one line' version of the client's X.509 certificate.
#     Note that no password is obtained from the user. Every entry in the user
#     file needs this password: `xxj31ZMTZzkVA'.
#   o ExportCertData:
#     This exports two additional environment variables: SSL_CLIENT_CERT and
#     SSL_SERVER_CERT. These contain the PEM-encoded certificates of the
#     server (always existing) and the client (only existing when client
#     authentication is used). This can be used to import the certificates
#     into CGI scripts.
#   o StdEnvVars:
#     This exports the standard SSL/TLS related `SSL_*' environment variables.
#     Per default this exportation is switched off for performance reasons,
#     because the extraction step is an expensive operation and is usually
#     useless for serving static content. So one usually enables the
#     exportation for CGI and SSI requests only.
#   o StrictRequire:
#     This denies access when "SSLRequireSSL" or "SSLRequire" applied even
#     under a "Satisfy any" situation, i.e. when it applies access is denied
#     and no other module can change it.
#   o OptRenegotiate:
#     This enables optimized SSL connection renegotiation handling when SSL
#     directives are used in per-directory context. 
#SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
    SSLOptions +StdEnvVars
<Directory "/var/www/cgi-bin">
    SSLOptions +StdEnvVars

#   SSL Protocol Adjustments:
#   The safe and default but still SSL/TLS standard compliant shutdown
#   approach is that mod_ssl sends the close notify alert but doesn't wait for
#   the close notify alert from client. When you need a different shutdown
#   approach you can use one of the following variables:
#   o ssl-unclean-shutdown:
#     This forces an unclean shutdown when the connection is closed, i.e. no
#     SSL close notify alert is send or allowed to received.  This violates
#     the SSL/TLS standard but is needed for some brain-dead browsers. Use
#     this when you receive I/O errors because of the standard approach where
#     mod_ssl sends the close notify alert.
#   o ssl-accurate-shutdown:
#     This forces an accurate shutdown when the connection is closed, i.e. a
#     SSL close notify alert is send and mod_ssl waits for the close notify
#     alert of the client. This is 100% SSL/TLS standard compliant, but in
#     practice often causes hanging connections with brain-dead browsers. Use
#     this only for browsers where you know that their SSL implementation
#     works correctly. 
#   Notice: Most problems of broken clients are also related to the HTTP
#   keep-alive facility, so you usually additionally want to disable
#   keep-alive for those clients, too. Use variable "nokeepalive" for this.
#   Similarly, one has to force some clients to use HTTP/1.0 to workaround
#   their broken HTTP/1.1 implementation. Use variables "downgrade-1.0" and
#   "force-response-1.0" for this.
SetEnvIf User-Agent ".*MSIE.*" \
         nokeepalive ssl-unclean-shutdown \
         downgrade-1.0 force-response-1.0

#   Per-Server Logging:
#   The home of a custom SSL log file. Use this when you want a
#   compact non-error SSL logfile on a virtual host basis.
CustomLog logs/ssl_request_log \
          "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b" 

SSLCaCertificatePath /home/sam/httpd/TrustedCAs
SSLCACertificateFile /home/sam/httpd/LocalCACertificates.pem

RequestHeader set X-Forwarded-Secure "ssl" 
RequestHeader set X-Forwarded-Proto "https" 
RequestHeader set X-FORWARDED-S-DN %{SSL_CLIENT_S_DN}s
RequestHeader set X-FORWARDED-I-DN %{SSL_CLIENT_I_DN}s

# Allow JSON requests from any domain
Header set Access-Control-Allow-Origin "*" 

SSLVerifyClient require
SSLVerifyDepth 15

RewriteEngine On


The items that are different from a standard SSL configuration:

Listen 8483

#   Some MIME-types for downloading Certificates and CRLs
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl
<VirtualHost _default_:8483>
SSLProtocol all -SSLv2 -SSLv3
Point your SSLCertificateFile and SSLCertificateKeyFile appropriately.  See section below to create a self-signed certificate.
SSLCaCertificatePath /home/sam/httpd/TrustedCAs
SSLCACertificateFile /home/sam/httpd/LocalCACertificates.pem

RequestHeader set X-Forwarded-Secure "ssl" 
RequestHeader set X-Forwarded-Proto "https" 
RequestHeader set X-FORWARDED-S-DN %{SSL_CLIENT_S_DN}s
RequestHeader set X-FORWARDED-I-DN %{SSL_CLIENT_I_DN}s

# Allow JSON requests from any domain
Header set Access-Control-Allow-Origin "*" 

SSLVerifyClient require
SSLVerifyDepth 15

RewriteEngine On

Generate a self-signed certificate for Apache (SSL)

sudo mkdir /etc/apache2/ssl
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/apache2/ssl/server.key -out /etc/apache2/ssl/server.crt
//then make it safe:
sudo chmod 600 /etc/apache2/ssl/*

SAM Admin Link

Some GIT Commands

To get the status of the repository.

git status

To exclude files from being tracked.
vim .git/info/exclude

To ignore files.
vim .gitignore

To see the branches.
git branch -a

To switch to a new branch named browser.
git checkout -b browser

To checkout master.
git checkout master
git pull

To see what the repository looks like.
git log --decorate --graph --all

To see things in color.
git log --decorate --graph --all --color=auto

To configure GIT with the basics.
vim ~/.gitconfig

        name = Michael Gheith
        email =


Find Your Running Processes

ps -fu <username>

Make API Call

Directly to CherryPy:

curl http://localhost:20020/sam/samdev/api/users

To Apache Web Server (if it is running and is using proxy pass - as root do service httpd start | status):

curl localhost/sam/samdev/api/users

Side Note: If you are using curl and are passing vars, then be sure to quote what you are curling.

Or use your browser:

Instead of using curl, you can use your browser to make requests directly to the CherryPy instance, skipping Apache.
On your local terminal:

ssh -L 20020: <your user name>

Then navigate to the following URL:

More on the SAM API:

Check If Authenticated

Provided you have a valid client side certificate in your browser:

Set Memory Limit

If you need to prevent your fermicloud instance from crashing because of memory related issues, then you can use ulimit. You would need to do this for every shell.

ulimit -a
ulimit -v 1500000

Point samweb Client To Your samweb Server



export SAM_WEB_BASE_URL=http://localhost/sam/samdev/api

For secure connections so you can declare files and the like:

export SAM_WEB_BASE_SSL_URL=https://localhost:8483/sam/samdev/api

Check that you can get authenticated by the server using SSL:

samweb -e samdev server-info -s

Apache Logs


Find Listening Ports

netstat --tcp --listen  --numeric-ports
netstat -tulpn

Connect Using Psql

psql -h cspgsdev -d samdev -p 5433

Python Requests

Here's an example of hitting a HTTPS API endpoint using the Python Requests library provided you retrieved your client side certificate by running kx509:

import requests
r = requests.get('', cert=('/<somepath>/x509up_u#####'), verify=False)

This is an example of posting JSON to a HTTPS endpoint:

import json
url = ''
headers = {'Content-Type': 'application/json'}
payload = {'removegroup': 'holometer'}
r =, data=json.dumps(payload), cert=('/<somepath>/x509up_u#####'), verify=False, headers=headers)