History

Installation and configuration procedure for Remote Operation Centers for the MINOS+ Experiment¶

This wiki describes the necessary steps to set up a full remote MINOS+ shift station. If your Institution already has a special kerberos principal, you can skip section 1.

Contents¶

Authentication and connectivity: explains how the connection with the Control Room is made and what a new ROC needs to do to be able to connect with it.
The Software: describes the necessary software for a remote shift and gives a brief description of the software package content provided for remote centers.
Installing the software: shows the necessary steps to configure a new ROC.
Configuring a new RMS: shows the necessary steps to configure a Remote MINOS+ Station.
Installing a VPN: Nonger necessary to access the MCR and NuMI E-logs.
Running the RMS softwares: shows how to use the software package.
List of Kerberos Principals and ports currently being used

1. Authentication and Connectivity¶

Most applications (whether they are running remotely or at Fermilab) connect to their servers over a port forwarded through a SSH tunnel. In order to have the connection, the remote station must fullfill two main conditions: presenting a valid Kerberos ticket and not conflicting with other allocated ports on a shared SSH gateway.

Get a Kerberos Services Principal for your station
- To get a valid Kerberos ticket one will need a Special Kerberos principal. For more information about this, visit
  Rules and Procedures for Using Special Kerberos Principals
- A Special Kerberos principal uses a keytab file instead of a password to provide the valid tickets. This keytab file must be on [rms-folder]/krb5/file.keytab

Send email to minos-run asking to have your principal registered.
- Someone with access to the minos account will update the .k5login file on the machines listed below, adding a new line with the new Special Kerberos account. The Kerberos Principals found in the .k5login file are shown at the end of this documentation.

SYSTEM	ROLE
minos-gateway-nd.fnal.gov	ND OM and RC
minos-gateway-fd.fnal.gov	FD OM and RC

2. The Software¶

MINOS+ Control Room was build to run under the Scientific Linux Fermi 5 in the 32 bit version. This should be the ROC operating system as well and it can be downloaded at http://fermilinux.fnal.gov.

It is strongly recommended that the new ROC uses at least two computers, each one powerful enough to run all the software simultaneously. This allows the shifter to split the load on more than one machine. Also, a second computer is important for redundancies: in case one computer crashes, there is still one working and capable of doing all the shift tasks.

MINOS+ shifts depends on various network based applications that monitor different parts of the experiment. Some of these are web applications

ECL, MemoPad & Elog: logs for reporting the status of the experiment and any issues related to it
Detector Conrtol System Web Pages (DCS): Monitors environmental conditions for the ND and the FD
Beam Monitoring: A9 monitor (logging status), Alarms, Numimon plots

while others require execution through an SSH tunnel

Run Control GUI (RC): Monitor and controls the DAQ running, for both ND and FD;
Online Monitoring (OM): Monitor the DAQ output for ND and FD;

These non-web applications are available in a tar file, which can be downloaded here

https://cdcvs.fnal.gov/redmine/attachments/download/23349/rms-20150213.tgz

When untarred, this will produce a directory with the following content:

desktop_launchers/ desktop launchers for the RCs, OMs and renewing tickets. They can be copied to the Desktop/ area and they replace the terminal command lines for using the rms script
om/ provides the OM binaries
mcr/ provides the RC binaries
root/ is a copy of ROOT v5.16, needed for the OM
rcroot/ is a copy of ROOT v.5.34, needed for the RC GUI
sys/ has some SLF 5.5 system libraries that are needed for the RC and OM
rms/ where the main script (rms) is. It will be explained with more details later

3. Installing the software¶

The software can be installed in any desired path. The rms main folder can be renamed if needed.

Unpack the software from the tar file:
- curl -o rms-20150213.tgz https://cdcvs.fnal.gov/redmine/attachments/download/23349/rms-20150213.tgz
- tar -xzvf rms.tgz

Copy your keytab file to [rms-folder]/krb5/file.keytab

For the 5.5 32 bits SLF version the software should work out of the box.

3.1 For SLF 6+ 64 bits¶

Even though the MINOS softwares were build to run under a 32 bit SLF, the new Control Room (a.k.a. ROC-West) at Fermilab uses SLF 6.5 in its 64 bits version. In this case, a set of libraries must be checked in order to make the RC Gui to run properly. The list of necessary libraries that may not be included in the standard installation of the SLF 6+ 64 bits is:

libpopt.so.0
libstdc++.so.6
libcrypto.so.6
libssl.so.6
libfreetype.so.6
libsigc-1.2.so.5
libssl.so.4
libcrypto.so.4
libXpm.so.4
libXft.so.2
libGLU.so.1
libjpeg.so.62
libpng12.so.0

Adam S.: I also noticed problems with running the OM on the machine I was setting up. There is an additional library needed.

libungif.so.4

These libs can be installed by running yum install [libname].

The Online Monitoring has bigger issues and, since it will be replaced by a web-based version soon, the current solution for the next few weeks is to do an ssh and run it from the minos-gateway-nd.fnal.gov machine. This can be done by doing:

ssh minos@minos-gateway-nd.fnal.gov
cd rms-20130910/rms/
./rms service om near

4. Configuring a new RMS¶

Every RMS (Remote MINOS+ Station) will need a config file that has to be created according to the following path: [rms-folder]/rms/config/`hostname -f`/config.
This config file defines the required variables used by the ROC:

RMS_RCOM_PORT_OFFSET: sets an offset used to calculate a unique port for RC/OM forwarding and it should be a small number
RMS_KEYTAB_FILE: sets the keytab file to use. Needed if "rms kinit" is used.
RMS_KEYNAME: sets the key name to use. Also needed if "rms kinit" is used.
RMS_PHONE_NUMBER: sets the primary phone number at which the primary phone in the shift room can be reached. Needed to simply "rms announce"

Example of a config file at [rms-folder]/rms/`hostname -f`/config¶

# CONFIG FILE EXAMPLE
#---------------------
RMS_RCOM_PORT_OFFSET=9
RMS_KEYTAB_FILE="$main_path/krb5/minos-om.keytab" 
RMS_KEYNAME="minos-wh-cr/minos/minos-om.fnal.gov@FNAL.GOV"  

# Remote Station's phone number:
#-------------------------------
RMS_PHONE_NUMBER="800 555 5555"

Configuring your own port numbers:
- HOST=`hostname -f`
- cd [rms-folder]/rms
- setting your keytab path and port number offset (see list at the end of this page)
  - mkdir config/${HOST}
  - cp config/minos-acnet.fnal.gov/config config/${HOST}/config
  - cp -r config/minos-acnet.fnal.gov/rcgui config/${HOST}/rcgui
  - nano config/${HOST}/config # or use your favorite editor
    - RMS_KEYTAB_FILE="$main_path/krb5/minos-om.keytab"
    - RMS_RCOM_PORT_OFFSET=9 # set your correct offset

*OBSOLETE - USE WEB NUMIMON * For NumiMon (JAS) callbacks, a port must be registered with iptables on minos-om, using the root account.
- edit /etc/sysconfig/iptables to add a line with the new port (19882 in this example):
```
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 19882 -j ACCEPT
```

The port allocation is organized through the file [rms-folder]/rms/config/`hostname -f`/config.
- All the used ports are listed below to ensure that one's choice will not confict with already estabilished ports.
  - This wiki is the official centrally maintained list of ports. They are listed at the end of this documentation.

In addition you can copy and customize the ND and FD RC configuration files: [rms-folder]/rms/config/`hostname -f`/rcgui/[near|far]/gui.config.
Example copies can be found in the directories for pre-existing sites. All that needs modification is the default RC connection labels at the top of each gui.config file by setting the site's contact information in the line guiLocation#S= ... ;. This save some effort each time an RC is started and it guarantee that the window title will be set correctly.

If your ROC has two or more computers, repeat the process for each remote station available.

5. Installing a VPN¶

The Accelerator Division Elog pages are available to anyone with a Fermilab account, using their Services password.
This is the account and password used to log into the experiment's ECL logbook and Redmine.
The new URL is https://www-bd.fnal.gov/Elog

Before 2013-07-22, the older AD elogs were restricted to fnal.gov addresses.

Obsolete VPN instructions, for reference¶

In order to install a VPN on SLF 5.5, open https://vpn.fnal.gov on your browser and follow the onscreen instructions.

If you receive the following error "Failed to install AnyConnect VPN Profile because of file move error. A VPN connection cannot be established.", copy the Fermilab.xml file (see below) to the /opt/cisco/anyconnect/profile/ folder and try again.

https://cdcvs.fnal.gov/redmine/attachments/download/10426/Fermilab.xml

6. Running the RMS software¶

All required software runs via the [rms-folder]/rms/rms script. This script takes a command as its first argument and possibly some specific options. Running the script with no options (or with the command help) will print some guidance.

In a normal shift, you will do :

rms kinit
rms service rc near
rms service rc far
rms service om near
rms service om far

This is the list of commands:

Set up commands¶

host: print what host names are beeing used
- Command: ./rms host [om|rc] [near|far]

port: print the used port numbers
- Command: ./rms port [om|rc] [near|far]

tunnel: forward a localport to a host through a gateway
- Command: ./rms tunnel PHP GW
  - where PHP = localport:host:port and GW = gateway

Shifter's commands¶

kinit: renew the Kerberos tickets
- Command: ./rms kinit

service: start an RC or OM application
- Command: ./rms service [om|rc] [near|far]
announce: display a message on minos-om indicating that a remote shift is in progress
- Command: ./rms announce [your name|kill]
- Important to say that this will display a default message based on the name given on the command line and the number defined on the variable RMS_PHONE_NUMBER. The command will connect on minos-om and the message will be displayed via xmessage. The kill command will log in and remove the message.

7. List of Kerberos Principals and ports currently being used¶

Current Kerberos Principals on MINOS gateway machines (2015-03-12)¶

Kerberos Principals listed in the .k5login file in minos-gateway-nd.fnal.gov and minos-gateway-fd.fnal.gov machines.

minerva-online/minerva/minerva-om.fnal.gov@FNAL.GOV
minerva-roc/minerva/23-25-1-49-static.hfc.comcastbusiness.net@FNAL.GOV
minerva-roc/minerva/ksmcf-cart.pas.rochester.edu@FNAL.GOV
minerva-roc/minerva/minerva-cart.phyast.pitt.edu@FNAL.GOV
minerva-roc/minerva/uroc02.phy.tufts.edu@FNAL.GOV
minerva-roc/minerva/uroc.d.umn.edu@FNAL.GOV
minerva-roc/minerva/uroc.fis.utfsm.cl@FNAL.GOV
minerva-roc/minerva/uroc.hep.utexas.edu@FNAL.GOV
minerva-roc/minerva/uroc.physics.wm.edu@FNAL.GOV
minerva-roc/minerva/uroc.phy.tufts.edu@FNAL.GOV
minerva-roc/minerva/uroc.wm.edu@FNAL.GOV
minerva-roc/minerva/photon.otterbein.edu@FNAL.GOV
minerva-roc/minerva/uroc.fisica.pucp.edu.pe@FNAL.GOV
minerva-roc/minerva/uroc.physics.oregonstate.edu@FNAL.GOV

minos-nd-cr/minos/minossrv-nd.fnal.gov@FNAL.GOV
minos-wh-cr/minos/minos-crl-wh.fnal.gov@FNAL.GOV
minos-wh-cr/minos/minos-om.fnal.gov@FNAL.GOV
minos-wh-cr/minos/minos-rc.fnal.gov@FNAL.GOV

rms/minos/ark.fnal.gov@FNAL.GOV
rms/minos/fuw.edu.pl@FNAL.GOV
rms/minos/goias.ufg.br@FNAL.GOV
rms/minos/spa.umn.edu@fnal.gov
rms/minos/ucl.ac.uk@FNAL.GOV
rms/minos/ph.utexas.edu@FNAL.GOV

Port offsets (`RMS_RCOM_PORT_OFFSET`) (2015-03-12)¶

1	Soudan Control Room
2	FNAL Control Room - minos-om
3	BNL Remote Shift Station
4	William and Mary UROC
5	Tufts UROC
6	Texas UROC
7	USM UROC
8	Rochester UROC
9	MINOS RMS Test Machine
10	MINERvA FNAL Control Room
11	Federal University of Goias
12	Univ Minnesota Duluth UROC
13	MINOS-SRV at Near Detector
14	Univ Pittsburgh UROC
15	Univ of Warsaw
16	Tufts Minos
17	minos-acnet
18	minos-evd
19	minos-rc
20	Univ Minnesota TC UROC
21	Tufts Minerva UROC
22	University College London
23	Pontificia Universidad Católica del Perú
24	Oregon State University
25	UT Austin

Development notes¶

Need to purge references to rms.sh in rms and documents

Need rms version command.

GIT - have started committing the RMS scripts to GIT under this Redmine project.

Files

Project

General

Profile

MINOS » RMS

Wiki

Installation and configuration procedure for Remote Operation Centers for the MINOS+ Experiment¶

Contents¶

1. Authentication and Connectivity¶

2. The Software¶

3. Installing the software¶

3.1 For SLF 6+ 64 bits¶

4. Configuring a new RMS¶

Example of a config file at [rms-folder]/rms/`hostname -f`/config¶

5. Installing a VPN¶

Obsolete VPN instructions, for reference¶

6. Running the RMS software¶

Set up commands¶

Shifter's commands¶

7. List of Kerberos Principals and ports currently being used¶

Current Kerberos Principals on MINOS gateway machines (2015-03-12)¶

Port offsets (`RMS_RCOM_PORT_OFFSET`) (2015-03-12)¶

Development notes¶

Project

General

Profile

MINOS » RMS

Wiki

Installation and configuration procedure for Remote Operation Centers for the MINOS+ Experiment¶

Contents¶

1. Authentication and Connectivity¶

2. The Software¶

3. Installing the software¶

3.1 For SLF 6+ 64 bits¶

4. Configuring a new RMS¶

Example of a config file at [rms-folder]/rms/`hostname -f`/config¶

5. Installing a VPN¶

Obsolete VPN instructions, for reference¶

6. Running the RMS software¶

Set up commands¶

Shifter's commands¶

7. List of Kerberos Principals and ports currently being used¶

Current Kerberos Principals on MINOS gateway machines (2015-03-12)¶

Port offsets (RMS_RCOM_PORT_OFFSET) (2015-03-12)¶

Development notes¶

Port offsets (`RMS_RCOM_PORT_OFFSET`) (2015-03-12)¶