Project

General

Profile

FTS for the Laser Scanner » History » Version 9

« Previous - Version 9/11 (diff) - Next » - Current version
Andrew Norman, 10/30/2012 05:05 PM


FTS for the Laser Scanner

The file transfer service has been setup as a transport layer for data taken with the laser scanner at Ash River. Setting up the system required some additional steps to make it possible to bridge the gap between the machine that runs the laser scanner software (which is a windows 7 laptop) and the linux based system that runs the file transfer service.

General Topology and Dataflow

We have setup the FTS system in a slightly different manner for use with the laser scanner than we have with the rest of the DAQ. The data is generated on a windows based pc that is connected to a laser scanner. The data is then processed locally on the windows machine to generate a set of files for further analysis. These files are bundled into a "tar" format archive file and placed in the c:\Export directory of the windows machine.

The machine running the FTS instance is then setup to mount the c:\Export directory from the windows machine, and performs a synchronization of the files that it finds in that area with another directory that resides on the FTS machine's local disk array. The area that is sync'd to is currently setup as /assembly which is a symbolic link to /data-b/assembly (a directory on the secondary raid array of the disk).

This area (/assembly) is set as the dropbox location that the FTS system will look in for new files.

When a new file is found it is transfered via an FTS relay (being hosted from novasamgpvm01.fnal.gov) to an intermediate area on the Fermilab bluearc (/nova/ana/assembly_fts). This area is set as the drop box for the relay FTS agent which then transfers the files to both the enstore tape system and to a well organized area on bluearc designed to be used for analysis (/nova/ana/assembly_ana). In these areas the files are organized by directories corresponding to the numbers of the blocks and then again in directories corresponding to the layers numbers.

External Mount

In order to mount the disk from the machine that is taking the laser scanner data the following steps were taken:

  • The cifs file system extensions had to be installed (part of base Linux install on this machine)
  • Optionally the cifs-utils package can be installed to aid with debugging (this was done on novadcs-far-master)

The file system is mounted using the command:

mount -t cifs //198.124.69.26/Export -o ro,username=LAZER\ SCANNER,password=THEPASSWORD,forceuid,forcegid,uid=novaraw,gid=nova /mnt/laser_scan

Note: You must use the ip address (not the network name) in the mount command in order for the windows system to correctly recognize this. Also, in order for the files in the volume to have the proper uid/gid combinations (so we can do things with them) we force a mapping to novaraw.nova so that the scripts that run the sync can be run as novaraw and do the right thing under rsync.

Now because we are working with a machine that lives on a different network (and can go up and down due to users cycling it) we need a way to make sure that if the mount is lost, we can restore it. To do this we have a small script which is run from the crontab which checks if the volume is mounted and if it isn't, mounts it.

The script is called "mount-laser-scanner.sh" and must be run from root's crontab in order to mount the disk.

The script currently reads:

#!/bin/bash
# This script ensures that the volume containing the 
# laser scanner data is mounted by the machine that 
# is running the FTS system
#
MOUNTPOINT=/mnt/laser_scan
SOURCEIP=198.124.69.26
SOURCEVOL=Export
USERNAME="LAZER SCANNER" 
PASS=CENSORED
DATE=date
HOSTNAME=hostname
MOUNT_UID=novaraw
MOUNT_GID=nova
/bin/mountpoint -q $MOUNTPOINT
if [ $? -ne 0 ]; then 
  mount -t cifs //${SOURCEIP}/${SOURCEVOL} -o ro,username=LAZER\ SCANNER,passwor
d=${PASS},forceuid,forcegid,uid=${MOUNT_UID},gid=${MOUNT_GID} ${MOUNTPOINT}
  if [ $? -ne 0 ]; then
    echo Failed to mount $MOUNTPOINT on $HOSTNAME at $DATE
  else
    echo Remounted $MOUNTPOINT on $HOSTNAME at $DATE 
  fi
#else
# echo Already mounted /mnt/laser_scan 
fi

Root's crontab is then setup as:

#
MAILTO=anorman@fnal.gov
#
# Run every 10 Minutes to restore mount points
*/10 * * * * /root/bin/mount-nova-disks.sh
*/10 * * * * /root/bin/mount-laser-scanner.sh

So that it checks every ten minutes to make sure that the mount is up.

Sync from Laser to Local

The sync from the (remote) Laser scanner machine to the local disk on the FTS machine is accomplished through a simple rsync. The script is located in the bin directory of the novaraw user on novadaq-far-datadisk-03. The script uses a VERY simple locking mechanism to ensure that rsyncs don't pile up. The script reads as:

#!/bin/bash
#
# This is a script that synchronizes the data 
# that is stored on the laser scanner machine (windows 7)
# to an area on the data disk that is watched by the FTS system
#
# This script is designed to be run via a crontab
SRC_TREE=/mnt/laser_scan/
DEST_TREE=/assembly/

LOCKFILE=/tmp/lazer-scanner-copy-lock
# Take out a lock so that we don't get multiple rsyncs running at
# the same time
if [ -f $LOCKFILE ]; then
  # A lock exists so we check the age and clear it
  # if it is stale
  echo Lock Exists
else
  # No lock exists so we take one out
  touch $LOCKFILE
  # Now we perform our sync
  rsync -av ${SRC_TREE} ${DEST_TREE}
  # Clean up our lock file
  rm $LOCKFILE
fi

The novaraw account then has a crontab that reads:

#
#MAILTO=anorman@fnal.gov
#
# Run every 4 hours to sync data from the laser scanner machine
# to this machine for the FTS
30 */4 * * * /home/novaraw/bin/sync_laser_scanner_data.sh

Which triggers an rsync every 4 hours between the areas.

FTS instance

After the files are copied into the FTS dropbox, they are handed over to the FTS system for transfer, validation, deletion. The status of files which are under FTS control can be found at:

http://novadaq-far-datadisk-03.fnal.gov:8888/fts/status

The FTS instance is installed under the novaraw user and is setup in the standard way. The instructions for starting/stopping the FTS are found on the main FTS page.

Trouble Shooting

The following items should be checked for if there is a suspected problem with the file transfers from the laser scanner:

  • Is the c:\Export area from the laser scanner machine mounted?

If the area is not mounted:

  • Remount the area by hand (note any error messages)
  • Verify that the windows machine is at the expected IP address (the raw IP address must be used in the mount command)
  • Verify that there is no firewall enabled on the windows machine that is rejecting the connection
  • Verify that the user name and password spell "laser" incorrectly with a Z (i.e. "Lazer")

Next:

  • Is the script that performs the rsync in the right location and set to go from the crontab file at the appropriate times (i.e. every 4 hours)

If it does not appear to be running:

  • Is there a stale lock file in the /tmp directory. If there is remove it manually.
  • Are the permissions on the src or destination directories incorrect (such that they would cause a copy of a file from one to the other as the user "novaraw" to fail. If this is the case change the ownership appropriately.

Next:

  • Is the FTS system running? (check the web based status page)
    • Not running? Restart it.
    • Still not happy? Check the status of the SAM server?
    • Still not happy? Call an expert.

A.Norman ()