Project

General

Profile

Tunnel monitoring » History » Version 2

« Previous - Version 2/6 (diff) - Next » - Current version
Matthew Tamsett, 04/05/2016 08:20 AM
Add in self links


Tunnel monitoring

This page explains the ROC tunnel monitor page.

Introduction

NOvA ROCs have to connect to VNC sessions in order to control the NOvA detectors. These VNC sessions live on machines that are only accessible within the fnal.gov domain. To connect to these machines remotely one must connect via gateway machines as explained on this page. This page documents the software used to monitor the status of these tunnels.

The ROC tunnel monitor software is provided in the devs repository here. It consists of a light weight monitoring client that runs on each of the two gateway nodes. The results of this monitoring are sent to a server that logs this information. Both the clients and the server are run via crontabs. Finally a web page visualises this information.

Common tools

The monitoring clients and server are both written in python and use a common set of tools provided here.

Client

The tunnel monitoring client runs on both of the gateway machines (novadaq-far-gateway-01, novadaq-near-gateway-01). The software to run it is installed here:

/home/novadaq/DAQ-gateway/monitoring

This software consists of a clone of the devs repository mentioned above. The software was manually put in place rather than checked out via SVN, this should be corrected. The client software can be run via the following commands:

$ cd /home/novadaq/DAQ-gateway/monitoring
$ ./run_tunnel_monitoring.sh 

This will result in the output:

--- Run status monitoring
Tue Apr  5 07:51:17 CDT 2016
--- user: novadaq
--- whoami: novadaq
--- Running
tools:     machine: novadaq-far-gateway-01.fnal.gov
tools:     2016-04-05 07:51:17
5900 5951 novacr01 novadaq-far-master-02 novacr01 Apr04 1:56
5901 5952 novacr02 novadaq-far-master-02 novacr02 Apr04 1:39
--- done
Tue Apr  5 07:51:17 CDT 2016

The above command runs a very simple wrapper script run_tunnel_monitoring.sh, which in turn runs the python script tunnel_monitoring_client.py with no augments. This python script run without any command line augments will determine the status of the tunnels using the functions in the common tools. Specifically it runs the FindTunnels function which uses the subprocess python module to run the ps aux shell command. The output of this command is parsed to find ssh tunnel processes. The details of any processses found are stored in an array of dictionary objects and then transmitted as a string via the UDP protocal to a server running on the machine and port configured in the script. Note the machine and port can be overridden using the "-a" and "-p" command line options to tunnel_monitoring_tools.py script if desired.

The client can also be used to send a few other simple commands to the monitoring server. These include a simple heartbeat message:

$ python tunnel_monitoring_client.py -b

a custom message:

$ python tunnel_monitoring_client.py -m "my message" 

or a shutdown command:

$ python tunnel_monitoring_client.py -s

This message will result in the termination of the server.

Server

Crontab control

Web page

To do

  • Replace manual copies of software on each of the gateway machines with SVN versioned software. This was not done intially as neither SVN or NOvA-soft is provided out-of-the-box on these machines.
    h2. Web page