Project

General

Profile

Design notes for support of Remote Control Room

Adding new control rooms

To add new remote controls rooms to the DAQ see:

Adding Remote Control Rooms

New functionality

The existing Control Room VNC utilities assume that the localhost is inside the fnal.gov domain. This will not be true for the case of remote control rooms, requiring a set of intermediate ssh tunnels to allow users outside the domain to connect to the VNC servers.

Outline

A machine inside the fnal.gov domain, but visible to the outside world, will need to serve as a gateway. For each VNC server in use in the control room system, there will need to be an ssh tunnel running on the gateway.

Requirements

Software functionality is needed to start, monitor, and stop the tunnels on the gateway machine, while avoiding port conflicts. Kerberos authentication must be used, in a way that minimizes risk of unauthorized access to special use principle keytabs.

Control Room Keytabs

Each control room is assigned a specific kerberos identity of the form:

nova-controlroom-<location>/

Where the <location> corresponds to the control room's location (e.g. harvard, ashriver, umn).

To obtain a valid ticket using this identity you use the kinit utility:

kinit -k -t <keytab filename> <kerberos identity>

Example:

kinit -k -t nova-controlroom-harvard.keytab nova-controlroom-harvard/nova/nova-daq-01.fnal.gov@FNAL.GOV

Once you have your ticket you can login to the NOvA gateway machines and VNC servers. The control room kerberos principles do NOT allow you access beyond the gateway or VNC server (i.e. to actual DAQ machines).

Details

VNCViewers running inside the fnal.gov domain simply connect to a port on the localhost that is tunneled via ssh to the VNCServer port on the remote node. For users running VNCViewers outside the fnal.gov domain, there will need to be an intermediate ssh tunnel connecting a port on the gateway to the VNCServer port. When those users start a VNCViewer, they will "double tunnel", establishing a tunnel from their localhost to the local port in the intermediate gateway tunnel. Since an intermediate tunnel will be needed for each of the ~dozen VNCViewers, utilities to stop, start, and monitor the intermediate tunnels will be needed.

Ash River Gateway

The gateway for the Ash River computing cluster is "novadaq-far-gateway-01.fnal.gov". This machine allows ssh connections from most places (Universities and Labs). We setup static tunnels on the gateway that map into the actual control room connections in the following manner:

# RUN ON NOVADAQ-FAR-GATEWAY-01
ssh -L 5951:localhost:5951 -N -f -l novacr01 novadaq-far-master-02.fnal.gov
ssh -L 5952:localhost:5952 -N -f -l novacr01 novadaq-far-master-02.fnal.gov
ssh -L 5953:localhost:5953 -N -f -l novacr01 novadaq-far-master-02.fnal.gov
ssh -L 5954:localhost:5954 -N -f -l novacr01 novadaq-far-master-02.fnal.gov

These setup the proper forwarding of ports between the VNC sessions (display 1=5951, display 2=5952, ....) and exposed port numbers.

Connecting from Onsite

If you are on the Fermilab site (or using the Fermilab VPN) then you can bypass the far detector gateway and connect to the hosts using the following template:

ssh -L <port on my machine>:localhost:<port on remote machine> -N -f -l <account name> <hostname>

Example for a standard VNC server running on port 5951 of novadaq-far-master-02.fnal.gov:

ssh -L 9999:localhost:5951 -N -f -l novacr01 novadaq-far-master-02.fnal.gov

Then connect to "localhost:9999" from your RealVNC client.

Connecting from Offsite

If you are connecting from offsite then you need to hop through the gateway. Assuming that the tunnels are in place you will do:

ssh -L 9999:localhost:5951 -N -f -l novacr01 novadaq-far-gateway-01.fnal.gov
vncviewer -Shared localhost:9999

This will actually map you from mymachine:9999 --> gateway:5951 --> master-02:5951

Gateway ports

The simplest scheme to assign local port numbers for the intermediate gateway tunnels would be to dead-reckon them to match the localhost port convention described here. However, this would only be guaranteed to work if the gateway machine is dedicated to NOvA Control Room VCN sessions, and no users or processes try to use ports in this range.

A more robust system, that could work without such restrictions, is to have a more dynamic system. The first component of that would be a daemon to start needed ssh tunnels, checking for availability of ports as it starts each tunnel. A range of ports well outside the VNC range could be used, say, in the 45055-45677 range. Then, a script on the gateway machine can do a ps to return the correct port for a given detector name and display number.

The current scheme in the NovaControlRoom package (bin/RemoteCR) just dead-reckons with a port offset of 1100, with absolutely no error checking.

nova-daq-04

The two VNC sessions on nova-daq-04 host web cam displays. These are very useful, but can take a lot of bandwidth. We should discuss whether we want to include nova-daq-04 VNC connections in the remote control room scheme.

Level of sophistication and ease of use.

How fancy do we want this to be? Do we just want simple scripts to launch, kill, and display the ssh tunnels processes from an interactive terminal on the gateway machine? Do we want to add GUIs that display the status of the tunnels? I suspect we would find such a tool to be very useful the 4th or 5th time we debug a remote control room problem without one.

Packaging and distribution

Simple scripts should probably live in NovaControlRoom/bin/RemoteCR. If more complex, compiled software is needed, do we want this to be compilable outside the NOvA DAQ SRT system?

Gateway machine

For the Far detector there is a dedicated gateway machine that has been setup to allow access from authorized remote control room sites.

The machine is: novadaq-far-gateway-01.fnal.gov

Security

How do we authenticate to minimize the chance of unauthorized access?

  • Which special-use principle do we use to attach to the VNC servers? nova-controlroom?
  • How do we distribute the special-use principle keytab in a way that minimizes the risk of unauthorized use?
    • Expert-only installation?
    • A checklist for ensuring correct file and directory permissions are set?
    • A script? (Probably not...)

Needed improvements.

VNC Servers

  • Mix detectors in same viewer?
  • Smaller viewers - more than one detector visible at a time?
  • Different Servers for different partitions?

Port assignments/Configurations

  • Depending on the answers above, we may need a better assignment of ports. The current system is algorithmic, which makes the usage of port numbers less dense (and therefore inefficient). A more efficient scheme would be a simple map of Server to Port, which could be stored in a file in CVS.