Project

General

Profile

Gateways

This document details the design of the tools for off-site connections to control room VNC sessions.

Introduction

The NOvA DAQ systems are run by experts and shifters in the control rooms (CR). These CRs host VNC servers which allow remote experts/shifters in their "remote control rooms" (RCR) to interact with the DAQ. The CR VNC servers only accept connections from within the FNAL.GOV domain, meaning RCRs must connect via a gateway (GW) within the FNAL.GOV domain which itself accepts external connections. A schematic is shown in Figure 1.

[CR] --- [GW] --- [RCR]

Fig 1. Connections schematic. (Key: [] : machine, --- : connection).

This document presents a quick start guide for users intending to connect remotely and two alternative schemes for the configuration of connections, the first involves static connections (RECOMMENDED) on the GW, the second dynamic. The configuration of connections is intended to be managed by experts.

Quick start guide for users

Assuming an expert has setup the gateway machine users with correct permissions can simply connect from their laptop or RCR. If you encounter permissions problems contact Peter Shanahan and ask to be added to novadaq. First you should check out the CVS control room software package:

kinit
export CVSROOT=novacvs@cdcvs.fnal.gov:/cvs/nova
cvs -d $CVSROOT co NovaControlRoom/scripts

Then navigate to the directory you just checked out.

cd NovaControlRoom/scripts/

Then connect your vncviewer to the machine of your choice (options are NDOS-1, NDOS-2, NDOS-3, FD-1, FD-2, FD-3) using a command of the form:

python AttachVNCSession.py -d NDOS-1

The argument "-V" (capitol V) will attach in view only mode. If you are using chicken or a similar program rather than vncviewer appending the "-b" option will find the port you should connect to on the gateway machine and print it out, but won't try and establish the viewer. You can then paste this port into your chicken configuration.

Attaching a VNC session spawns a local tunnel, the user should destroy this using VNCPortForwarding when they're done:

 python VNCPortForwarding.py -r all 

Static gateway setup for experts (RECOMMENDED)

This method consists of two steps, firstly the gateway setup, then secondly the remote connection. Figure 2 shows a map of the final connection setup.

[CR - A] --- [GW - port X] --- [RCR - 1]
                           --- [RCR - 2]

[CR - B] --- [GW - port Y] --- [RCR - 1]
                           --- [RCR - 2]

Fig 2. Static gateway connection schematic.

Static gateway setup

In the static gateway setup a single ssh tunnel is first established on the GW for each CR VNC server. RCRs then all connect to this same tunnel. The tunnel is established via the use of commands on the gateway machines novadaq-near-gateway-01 or novadaq-far-gateway-01 for each desktop.

The connections are configured by the "connections" dictionary in the Connections.py file and consist of a series of keys corresponding the CR machine desktop aliases. The values which are a tuple of the CR machine a user and a port.

> tools: --- Establish SSH Tunnel

VNCPortForwarding calls methods from "Tools.py". The first such method will establish the SSH tunnel to the CR.
> tools:     target host:   nova03.fnal.gov
> tools:     target port:   5998
> tools:     local port:    0

A local port can be specified, if not we default to looking for an open port in the range 5900 to 6000 (configurable).
> tools:     starting port: 5900
> tools:     user:          tamsett

The username is provided for each CR in the displays dictionary in the Connections.py file.
> tools:        - checking status of port 5900

We now call "Tools.FindFreePorts" which finds the first free port in the specified range.
> tools:        - free port found: 5900

In this case the first port is open, so we use that.
> tools:        - establishing ssh tunnel

We now use the python subprocess module to spawn our ssh tunnel.
> tools:        - confirming tunnel established

Some simple error checking is run. Firstly, if ssh returns a non-zero exit code we abort. Secondly we check that the tunnel we tried to create does exist:
> tools: --- List SSH Tunnels, specific connection:  ('nova03.fnal.gov', 5998, 5900)

This uses subprocess to call "ps aux" and parses the output.
> tools:     OpenSSHTunnel:
> tamsett  23043  0.0  0.0  44428   724 ?        Ss   05:41   0:00 ssh -L 5900:localhost:5998 -N -f -l tamsett nova03.fnal.gov
> tools:     Matching SSHTunnel:
>        local port:  5900
>        remote port: 5998
>        user:        tamsett
>        machine:     nova03.fnal.gov
> tools:        - established ssh tunnel, pid: 23042

In this case we see that an open ssh tunnel process to our target CR machine exists, on the open free local port we specified, to the correct CR port. If this wasn't the case, we'd abort.
> VNCPF:     connection for host nova03.fnal.gov, remote port: 5998, mapped to local port: 5900

Success. VNCPortForwarding then repeats for each remaining connection (in this case one more is configured):
> VNCPF:     setting up connection Test-2 for host nova03.fnal.gov, remote port: 5999
> ...
> tools:        - free port found: 5901

Notice that as port 5900 is in use, we now use the next port.
    
> …
> VNCPF:     connection for host nova03.fnal.gov, remote port: 5999, mapped to local port: 5901
> VNCPF: done

The process has now completed. We can view our open tunnels with the command:
python VNCPortForwarding.py -l all [--verbose]

The non-verbose output shows as:
    
> 5900 5998 tamsett nova03.fnal.gov
> 5901 5999 tamsett nova03.fnal.gov

In this case we see our two open tunnels to the CR machine and ports. Tunnels can be destroyed on the gateway using:
python VNCPortForwarding.py -r all

Remote connection to static gateways

RCRs connect to established ssh tunnel on the GW and through that to the appropriate CR. Connections are established with:

    
python AttachVNCSession.py -d Test-1

The "detector" key supplied to the -d option must match with one in the "connections" dictionary, in this case "Test-1" is:
"Test-1"        : ("nova03.fnal.gov",5998)

Which corresponds to the first connection established in the static gateway setup stage. This results in the following output:
> AVNCS: --- AttachVNCSession:
> …
> AVNCS: --- Attaching VNC session for detector: Test-1, gateway: nova04.fnal.gov

The gateway machine defaults to novatest01.fnal.gov but can be changed with the -g command line option.
> AVNCS:     static mode, asking gateway for static tunnel port
> tools: --- FindGatewayTunnel, on host: nova04.fnal.gov to detector: Test-1

We then use Tools.FindGatewatTunnel to find out the port on the GW that is mapped to the CR and port we care about. This uses subprocess to spawn an ssh process which runs:
python ~/DAQ/VNCPortForwarding.py -l Test-1

On the gateway machine. Which, if there is an open connection, will stdout the port we should connect to. If no open connection is found or the ssh process has a non-zero exit code we exit.
> tools:     got remote port:  5900
> AVNCS:     creating local ssh tunnel to gateway port: 5900
> tools: --- Establish SSH Tunnel
> …
> AVNCS:     connection for gateway nova04.fnal.gov, remote port: 5900, mapped to local port: 5901

We then make a local ssh tunnel which will connect to the remote port. Finally the VNC viewer is booted.
    
> AVNCS:     Initalising VNC viewer

Alternative we can skip the VNC viewer starting and just stdout the local port we have used to establish our connection using the -b option:
python AttachVNCSession.py -d Test-1 -b

Viola. Additional options are provided for just finding the port on the GW we should connect to (-b) and for view only connection (-V).

Dynamic gateway setup (NOT RECOMMENDED)

This is a single step method in which the RCR requests to be connected to a given CR, an ssh tunnel is then spawned on the gateway and the RCR connects to that. The final connection schematic is shown in Figure 3.

[CR - A] --- [GW - port W] --- [RCR - 1]
             [GW - port X] --- [RCR - 2]

[CR - B] --- [GW - port Y] --- [RCR - 1]
             [GW - port Z] --- [RCR - 2]

Fig 2. Dynamic gateway connection schematic.

This is achieved via the use of the -D (dynamic) option to AttachVNCSession:

    
python AttachVNCSession.py -d Test-1 -D -v

This differs from the above usage in that rather than connecting to an established ssh tunnel we spawn our own over ssh:
> AVNCS: --- AttachVNCSession:
> …
> AVNCS: --- Attaching VNC session for detector: Test-1, gateway: nova04.fnal.gov
> AVNCS:     dynamic mode, asking gateway to set up a new tunnel
> tools: --- SetupGatewayTunnel
> tools: --- FindRemoteFreePort, host: nova04.fnal.gov

This uses Tools.SetupGatewayTunnel. The first stage of this is to find a free port on the GW machine. This done by sending an ssh command to the GW which calls:
python ~/DAQ/VNCPortForwarding.py -f

which prints out the first free port in the 5900 to 6000 range on the machine
    
> tools:     remote free port found: 5900
> tools: --- EstabishRemoteSSHTunnel, host: nova04.fnal.gov, detectorID: Test-1, remote port: 5900

We then establish an ssh tunnel on the GW to the desired CR who's local port (on the GW) is the free port previously identified. However doing this is tricky, as if we use ssh to spawn an ssh -L process we don't get a return. The process just hangs after completion. So instead we pass ssh the "-f" option, which starts running our command then exits. We then wait a few seconds for the process to complete and then confirm that it did. This is messy (using -tt option has the same behaviour with the added benefit of screwing with the terminal).
> tools: -------- spawned process, waiting for it to complete

The spawned process the runs on the GW during the wait time, the stdout is printed locally:
> VNCPF: --- VNCPortForwarding
> ...
> VNCPF:     connection for host nova03.fnal.gov, remote port: 5998, mapped to local port: 5900
> VNCPF: done

As we already exited the GW we cannot communicate with the spawned process. Meaning we have to include an additional step to confirm our tunnel spawning worked as desired. This is liable to cause bugs later on. Once we've waited a reasonable amount of time (10 seconds) we then confirm our remote tunnel exists:
> tools: -------- confirming process spawned
> tools: --- ConfirmRemoteSSHTunnel: nova04.fnal.gov:5900
> tools:     tunnel[0]:  5900
> tools:     confirmed

Assuming that it did we then create our local ssh tunnel and attach VNC to it as previously. If the tunnel is not confirmed we exit.

Notes and Questions

Q. Static tunnel setup has only been tested by myself. Can different users connect to tunnels I've setup?
A. Yes

Spawning ssh tunnels all over the place can cause VNCservers to start rejecting incoming request due to "too many security failures", want to avoid this!

Attaching a VNC session spawns a local tunnel, the user should destroy this using VNCPortForwarding when they're done.