Installation and configuration procedure for Remote Operation Centers for the MINOS+ Experiment¶
This wiki describes the necessary steps to set up a full remote MINOS+ shift station. If your Institution already has a special kerberos principal, you can skip section 1.
Contents¶
- Authentication and connectivity: explains how the connection with the Control Room is made and what a new ROC needs to do to be able to connect with it.
- The Software: describes the necessary software for a remote shift and gives a brief description of the software package content provided for remote centers.
- Installing the software: shows the necessary steps to configure a new ROC.
- Configuring a new RMS: shows the necessary steps to configure a Remote MINOS+ Station.
- Installing a VPN: Nonger necessary to access the MCR and NuMI E-logs.
- Running the RMS softwares: shows how to use the software package.
- List of Kerberos Principals and ports currently being used
1. Authentication and Connectivity¶
Most applications (whether they are running remotely or at Fermilab) connect to their servers over a port forwarded through a SSH tunnel. In order to have the connection, the remote station must fullfill two main conditions: presenting a valid Kerberos ticket and not conflicting with other allocated ports on a shared SSH gateway.
- Get a Kerberos Services Principal for your station
- To get a valid Kerberos ticket one will need a Special Kerberos principal. For more information about this, visit
Rules and Procedures for Using Special Kerberos Principals - A Special Kerberos principal uses a keytab file instead of a password to provide the valid tickets. This keytab file must be on
[rms-folder]/krb5/file.keytab
- To get a valid Kerberos ticket one will need a Special Kerberos principal. For more information about this, visit
- Send email to minos-run asking to have your principal registered.
- Someone with access to the minos account will update the .k5login file on the machines listed below, adding a new line with the new Special Kerberos account. The Kerberos Principals found in the .k5login file are shown at the end of this documentation.
SYSTEM | ROLE |
minos-gateway-nd.fnal.gov | ND OM and RC |
minos-gateway-fd.fnal.gov | FD OM and RC |
2. The Software¶
MINOS+ Control Room was build to run under the Scientific Linux Fermi 5 in the 32 bit version. This should be the ROC operating system as well and it can be downloaded at http://fermilinux.fnal.gov.
- It is strongly recommended that the new ROC uses at least two computers, each one powerful enough to run all the software simultaneously. This allows the shifter to split the load on more than one machine. Also, a second computer is important for redundancies: in case one computer crashes, there is still one working and capable of doing all the shift tasks.
- ECL, MemoPad & Elog: logs for reporting the status of the experiment and any issues related to it
- Detector Conrtol System Web Pages (DCS): Monitors environmental conditions for the ND and the FD
- Beam Monitoring: A9 monitor (logging status), Alarms, Numimon plots
- Run Control GUI (RC): Monitor and controls the DAQ running, for both ND and FD;
- Online Monitoring (OM): Monitor the DAQ output for ND and FD;
These non-web applications are available in a tar file, which can be downloaded here
https://cdcvs.fnal.gov/redmine/attachments/download/23349/rms-20150213.tgz
When untarred, this will produce a directory with the following content:
desktop_launchers/
desktop launchers for the RCs, OMs and renewing tickets. They can be copied to the Desktop/ area and they replace the terminal command lines for using the rms scriptom/
provides the OM binariesmcr/
provides the RC binariesroot/
is a copy of ROOT v5.16, needed for the OMrcroot/
is a copy of ROOT v.5.34, needed for the RC GUIsys/
has some SLF 5.5 system libraries that are needed for the RC and OMrms/
where the main script (rms) is. It will be explained with more details later
3. Installing the software¶
The software can be installed in any desired path. The rms main folder can be renamed if needed.
- Unpack the software from the tar file:
- curl -o rms-20150213.tgz https://cdcvs.fnal.gov/redmine/attachments/download/23349/rms-20150213.tgz
- tar -xzvf rms.tgz
- Copy your keytab file to
[rms-folder]/krb5/file.keytab
For the 5.5 32 bits SLF version the software should work out of the box.
3.1 For SLF 6+ 64 bits¶
Even though the MINOS softwares were build to run under a 32 bit SLF, the new Control Room (a.k.a. ROC-West) at Fermilab uses SLF 6.5 in its 64 bits version. In this case, a set of libraries must be checked in order to make the RC Gui to run properly. The list of necessary libraries that may not be included in the standard installation of the SLF 6+ 64 bits is:
libpopt.so.0
libstdc++.so.6
libcrypto.so.6
libssl.so.6
libfreetype.so.6
libsigc-1.2.so.5
libssl.so.4
libcrypto.so.4
libXpm.so.4
libXft.so.2
libGLU.so.1
libjpeg.so.62
libpng12.so.0
libungif.so.4
These libs can be installed by running yum install [libname]
.
ssh minos@minos-gateway-nd.fnal.gov
cd rms-20130910/rms/
./rms service om near
4. Configuring a new RMS¶
Every RMS (Remote MINOS+ Station) will need a config file that has to be created according to the following path: [rms-folder]/rms/config/`hostname -f`/config
.
This config file defines the required variables used by the ROC:
RMS_RCOM_PORT_OFFSET
: sets an offset used to calculate a unique port for RC/OM forwarding and it should be a small numberRMS_KEYTAB_FILE
: sets the keytab file to use. Needed if "rms kinit" is used.RMS_KEYNAME
: sets the key name to use. Also needed if "rms kinit" is used.RMS_PHONE_NUMBER
: sets the primary phone number at which the primary phone in the shift room can be reached. Needed to simply "rms announce"
Example of a config file at [rms-folder]/rms/`hostname -f`/config
¶
# CONFIG FILE EXAMPLE #--------------------- RMS_RCOM_PORT_OFFSET=9 RMS_KEYTAB_FILE="$main_path/krb5/minos-om.keytab" RMS_KEYNAME="minos-wh-cr/minos/minos-om.fnal.gov@FNAL.GOV" # Remote Station's phone number: #------------------------------- RMS_PHONE_NUMBER="800 555 5555"
- Configuring your own port numbers:
- HOST=`hostname -f`
- cd [rms-folder]/rms
- setting your keytab path and port number offset (see list at the end of this page)
- mkdir config/${HOST}
- cp config/minos-acnet.fnal.gov/config config/${HOST}/config
- cp -r config/minos-acnet.fnal.gov/rcgui config/${HOST}/rcgui
- nano config/${HOST}/config # or use your favorite editor
- RMS_KEYTAB_FILE="$main_path/krb5/minos-om.keytab"
- RMS_RCOM_PORT_OFFSET=9 # set your correct offset
- *OBSOLETE - USE WEB NUMIMON * For NumiMon (JAS) callbacks, a port must be registered with iptables on minos-om, using the root account.
- edit /etc/sysconfig/iptables to add a line with the new port (19882 in this example):
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 19882 -j ACCEPT
- edit /etc/sysconfig/iptables to add a line with the new port (19882 in this example):
- The port allocation is organized through the file
[rms-folder]/rms/config/`hostname -f`/config
.- All the used ports are listed below to ensure that one's choice will not confict with already estabilished ports.
- This wiki is the official centrally maintained list of ports. They are listed at the end of this documentation.
- All the used ports are listed below to ensure that one's choice will not confict with already estabilished ports.
In addition you can copy and customize the ND and FD RC configuration files: [rms-folder]/rms/config/`hostname -f`/rcgui/[near|far]/gui.config
.
Example copies can be found in the directories for pre-existing sites. All that needs modification is the default RC connection labels at the top of each gui.config file by setting the site's contact information in the line guiLocation#S= ... ;
. This save some effort each time an RC is started and it guarantee that the window title will be set correctly.
If your ROC has two or more computers, repeat the process for each remote station available.
5. Installing a VPN¶
The Accelerator Division Elog pages are available to anyone with a Fermilab account, using their Services password.
This is the account and password used to log into the experiment's ECL logbook and Redmine.
The new URL is https://www-bd.fnal.gov/Elog
Before 2013-07-22, the older AD elogs were restricted to fnal.gov addresses.
Obsolete VPN instructions, for reference¶
In order to install a VPN on SLF 5.5, open https://vpn.fnal.gov on your browser and follow the onscreen instructions.
If you receive the following error "Failed to install AnyConnect VPN Profile because of file move error. A VPN connection cannot be established.", copy the Fermilab.xml file (see below) to the /opt/cisco/anyconnect/profile/ folder and try again.
https://cdcvs.fnal.gov/redmine/attachments/download/10426/Fermilab.xml
6. Running the RMS software¶
All required software runs via the [rms-folder]/rms/rms
script. This script takes a command as its first argument and possibly some specific options. Running the script with no options (or with the command help) will print some guidance.
In a normal shift, you will do :
rms kinit rms service rc near rms service rc far rms service om near rms service om far
This is the list of commands:
Set up commands¶
- host: print what host names are beeing used
- Command:
./rms host [om|rc] [near|far]
- Command:
- port: print the used port numbers
- Command:
./rms port [om|rc] [near|far]
- Command:
- tunnel: forward a localport to a host through a gateway
- Command:
./rms tunnel PHP GW
- where
PHP = localport:host:port
andGW = gateway
- where
- Command:
Shifter's commands¶
- kinit: renew the Kerberos tickets
- Command:
./rms kinit
- Command:
- service: start an RC or OM application
- Command:
./rms service [om|rc] [near|far]
- Command:
- announce: display a message on minos-om indicating that a remote shift is in progress
- Command:
./rms announce [your name|kill]
- Important to say that this will display a default message based on the name given on the command line and the number defined on the variable
RMS_PHONE_NUMBER
. The command will connect on minos-om and the message will be displayed via xmessage. The kill command will log in and remove the message.
- Command:
7. List of Kerberos Principals and ports currently being used¶
Current Kerberos Principals on MINOS gateway machines (2015-03-12)¶
Kerberos Principals listed in the .k5login file in minos-gateway-nd.fnal.gov and minos-gateway-fd.fnal.gov machines.
minerva-online/minerva/minerva-om.fnal.gov@FNAL.GOV minerva-roc/minerva/23-25-1-49-static.hfc.comcastbusiness.net@FNAL.GOV minerva-roc/minerva/ksmcf-cart.pas.rochester.edu@FNAL.GOV minerva-roc/minerva/minerva-cart.phyast.pitt.edu@FNAL.GOV minerva-roc/minerva/uroc02.phy.tufts.edu@FNAL.GOV minerva-roc/minerva/uroc.d.umn.edu@FNAL.GOV minerva-roc/minerva/uroc.fis.utfsm.cl@FNAL.GOV minerva-roc/minerva/uroc.hep.utexas.edu@FNAL.GOV minerva-roc/minerva/uroc.physics.wm.edu@FNAL.GOV minerva-roc/minerva/uroc.phy.tufts.edu@FNAL.GOV minerva-roc/minerva/uroc.wm.edu@FNAL.GOV minerva-roc/minerva/photon.otterbein.edu@FNAL.GOV minerva-roc/minerva/uroc.fisica.pucp.edu.pe@FNAL.GOV minerva-roc/minerva/uroc.physics.oregonstate.edu@FNAL.GOV minos-nd-cr/minos/minossrv-nd.fnal.gov@FNAL.GOV minos-wh-cr/minos/minos-crl-wh.fnal.gov@FNAL.GOV minos-wh-cr/minos/minos-om.fnal.gov@FNAL.GOV minos-wh-cr/minos/minos-rc.fnal.gov@FNAL.GOV rms/minos/ark.fnal.gov@FNAL.GOV rms/minos/fuw.edu.pl@FNAL.GOV rms/minos/goias.ufg.br@FNAL.GOV rms/minos/spa.umn.edu@fnal.gov rms/minos/ucl.ac.uk@FNAL.GOV rms/minos/ph.utexas.edu@FNAL.GOV
Port offsets (RMS_RCOM_PORT_OFFSET
) (2015-03-12)¶
1 | Soudan Control Room |
2 | FNAL Control Room - minos-om |
3 | BNL Remote Shift Station |
4 | William and Mary UROC |
5 | Tufts UROC |
6 | Texas UROC |
7 | USM UROC |
8 | Rochester UROC |
9 | MINOS RMS Test Machine |
10 | MINERvA FNAL Control Room |
11 | Federal University of Goias |
12 | Univ Minnesota Duluth UROC |
13 | MINOS-SRV at Near Detector |
14 | Univ Pittsburgh UROC |
15 | Univ of Warsaw |
16 | Tufts Minos |
17 | minos-acnet |
18 | minos-evd |
19 | minos-rc |
20 | Univ Minnesota TC UROC |
21 | Tufts Minerva UROC |
22 | University College London |
23 | Pontificia Universidad Católica del Perú |
24 | Oregon State University |
25 | UT Austin |
Development notes¶
Need to purge references to rms.sh in rms and documents
Need rms version command.
GIT - have started committing the RMS scripts to GIT under this Redmine project.