Project

General

Profile

Test CERN VM-FS

Introduction

CernVM File System (CernVM-FS) is a network file system based on HTTP and optimized to deliver experiment software in a fast, scalable, and reliable way. Information for CVMFS can be found at http://www.gridpp.ac.uk/wiki/RAL_Tier1_CVMFS and http://cernvm.cern.ch/portal/ . This may be very useful for the Intensity Frontier experiments, for diverse clients and running processing at OSG GRID sites.

Test plan

We will use a VM on the fermicloud on which we install an Apache Tomcat server, and run the service. We may need a few hundred GB storage for this, probably this will be in IF Bluearc areas. Our general plan for this test is to:

1. set up the server side
2. load the repository with some small example
3. set up the FUSE module on a test client and confirm things work
4. review our test with the experiments and understand their interest.
(5). work with GRID and FEF about getting the FUSE module installed on the GP cluster
(6). proceed to production service, perhaps also on the fermicloud.

We would like to finish 3 in time to report NuComp meeting and begin getting experiment feedback. (5) and (6) are contingent on positive results for the other tasks.

Report and suggested setup

Why CVMFS?

NFS mounted BlueArc storage is currentlu employed for distributing experiment releases, and other input prosessing data for Intensity Frontier (IFront) experiments. A lightweight file system like CVMFS could potentially remove this load on the central storage service, as well as enable off site acces to these resources. In addition to software releases, configurations, and input to simulation have also represented a load on the central disks and been an impediment to offside code development, testing, and production. If the implementation of CVMFS can be operationally effective, and reliable, it may be a solution for many of these problems.

Overview

Installation and setup is straightforward and the steps are documented below (this can be streamlined). Two options were explored for the service, 1) sharing a single server by all experiments and 2) setting up a VM for each experiment with a dedicated service. There are implications for each scenario on both the server side and the client side. After trying both, it appears the service will work best if each experiment is assigned a server VM.

There are several parts that need to be understood in order to evaluate the system. The working release repository needs to be maintained in what is called the "shadow tree". This is a directory tree where the code, data, binaries, etc. are kept and initially tested for proper functionality when changes are made. The shadow tree is converted into a web-served set of files which are identified by their MD5 hash codes and managed with a set of nested SQLite databases that record the directory structure and point to individual files in the web repository. There is a server running on the server side that watches the shadow tree and records any changes. Any changes to the shadow tree must be made while the server is running. The web repository can be synchronized with the shadow tree by issuing a "cvmfs-sync" command and the following occurs: 1. the shadow file system is locked and no changes are allowed, 2. any changes in the change log are translated and added to the web catalog. Special tag files (.cvmfscatalog) in the shadow tree indicate where branches in the SQLite databases are created.

The web catalog is served with an Apache server. There is a public (pub) directory configured to be served, and an important redirect configuration that enables to client to properly access the catalogs in the pub area. The file system for the pub area cannot be the same as that of the shadow area since shadow is locked when the sync is initiated. The cvmfs-sync can be run by a non-root user, however the user must have read-write acces to both the shadow and pub directories.

On the client side, there are two ways to run the service. The easiest to get started is using the cvmfs2, or cvmfs2_debug commands with arguments that specify the appropriate http servers, proxy servers, cache area, mount points, and other parameters. Running the "standard" cvmfs client service requires that the catalog (on the server) be signed with a .x509 key and this key must be present for each client. There is a hiararchy of configurations that are maintained for the client system for general/default, domain specific, and local setup. It is possible to specify multiple servers and proxy-caching servers.

Shared service

In a shared service, many experiments would use a common machine and maintain their software repositories under a common shadow tree. Unfortunately, although it is possible to synchronize the pub (web catalogs) with the shadow tree as a user other than root, all directories in the shadow and pub areas need to be owned by the user running the sync. This issue is somewhat mitigated if the sync is done by root, although this is not a desirable configuration for operational and security reasons.

On the client side, one convenient feature is that the file system can be mounted at any level where a catalog (.cvmfscatalog) exists in the shadow tree. This would enable each experiment to mount just their portion of the directory tree on the client side. It enables any experiment to mount any other experiments directories, which may not be an issue as the client side FS is read only, and this is largely the existing situation with blue arc mounts anyway.

Dedicated services

The case of each experiment having their own service on a dedicated VM is a better choice. With this configuration, a single user (group shared) account controled by the experiment can be used to manage the shadow tree, and synchronize to the pub area. It still allows the client to mount at any point in hte shadow directory where a .cvmfscatalog tag is located. For example, if multiple release versions are provided any of them could be exclusively mounted on the client side.

Proposed setup

With this in mind, it is proposed that a production set up of CVMFS for the Intensity Frontier be based on the dedicated service model. Each experiment would have one or more VM servers. It is possible to use existing experiment code repositiries as the shadow area, as long as they have a single owner, mounted RW on the cvmfs server VM, and be allowed to have .cvmfscatalog tags inserted into them. The server components and httpd will be installed and maintained each VM as root. It is posisble to run the services (cvmfsd and httpd) as non-root user although they will be started in the init.d directory to start with the VM. These need to be available outside the fermilab domain.

  • Proposed server names
    1. minervafs01, minervafs02
    2. novafs01, novafs02
    3. gm2fs01, gm2fs02
    4. etc. for other Intensity Frontier groups, ~6 in number

Operation details

  • Tasks and responsibilities
    1. VM servers - Fermicloud
    2. Setup & maintain cvmfsd, httpd - REX
    3. Provide client installation instructions and configuration - REX
    4. Install and configure GP Grid cluster nodes - FEF + Fermigrid
    5. Provide client experiment specific configuration - REX + experiment
    6. Maintain repositories & sync catalogs - Experiment
    7. Provide client usage examples for users - Experiment

Specifics for client install and config

install

  1. yum install fuse
  2. wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-keys-1.1-2.noarch.rpm
  3. wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-2.0.3-1.x86_64.rpm
  4. wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-init-scripts-1.0.13-1.noarch.rpm
  5. rpm -ivh cvmfs-keys-1.1-2.noarch.rpm cvmfs-2.0.3-1.x86_64.rpm cvmfs-init-scripts-1.0.13-1.noarch.rpm
  6. cvmfs_config setup

config

  1. local conf
    [root@fermicloud048 ~]# cat /etc/cvmfs/default.conf
    # Don't edit here.  Create /etc/cvmfs/default.local.
    # As a rule of thumb, overwrite only parameters you find in here.
    # If you look for any other parameter, check /etc/cvmfs/domain.d/<your_domain>.(conf|local)
    # and /etc/cvmfs/config.d/<your_repository>.(conf|local)
    #
    # Parameter files are sourced in the following order
    # /etc/cvmfs/default.conf
    # /etc/cvmfs/default.local
    # /etc/cvmfs/domain.d/<your_domain>.conf
    # /etc/cvmfs/domain.d/<your_domain>.local
    # /etc/cvmfs/config.d/<your_repository>.conf
    # /etc/cvmfs/config.d/<your_repository>.local
    #
    # Use cvmfs_config showconfig to get the effective parameters.
    #
    
    CVMFS_CACHE_BASE=/var/cache/cvmfs2
    CVMFS_QUOTA_LIMIT=-1
    CVMFS_DEFAULT_DOMAIN=fnal.gov
    CVMFS_TIMEOUT=5
    CVMFS_TIMEOUT_DIRECT=10
    CVMFS_STRICT_MOUNT=yes
    #CVMFS_FORCE_SIGNING=yes
    
    # Don't touch the following values unless you're absolutely
    # sure what you do.  Don't copy them to default.local either.
    if [ "x$CVMFS_BASE_ENV" == "x" ]; then
      readonly CVMFS_USER=cvmfs
      readonly CVMFS_NFILES=32768
      readonly CVMFS_MOUNT_DIR=/cvmfs
      readonly CVMFS_OPTIONS=allow_other,entry_timeout=60,attr_timeout=60,negative_timeout=60,use_ino
      readonly CVMFS_BASE_ENV=1
    fi
    [root@fermicloud048 ~]#
    
    
  2. local defaults
    [root@fermicloud048 ~]# cat /etc/cvmfs/default.local
    CVMFS_REPOSITORIES=ifcvmfs.fnal.gov
    CVMFS_HTTP_PROXY="http://edge.fnal.gov:3128" 
    
  3. domain specific config
    [root@fermicloud048 ~]# cat /etc/cvmfs/domain.d/fnal.gov.conf
    # Don't edit here.  Create /etc/cvmfs/domain.d/cern.ch.local.
    # As a rule of thumb, overwrite only parameters you find in here.
    # If you look for any other parameter, check /etc/cvmfs/default.(conf|local)
    # and /etc/cvmfs/config.d/<your_repository>.(conf|local)
    #
    # Parameter files are sourced in the following order
    # /etc/cvmfs/default.conf
    # /etc/cvmfs/default.local
    # /etc/cvmfs/domain.d/<your_domain>.conf
    # /etc/cvmfs/domain.d/<your_domain>.local
    # /etc/cvmfs/config.d/<your_repository>.conf
    # /etc/cvmfs/config.d/<your_repository>.local
    #
    # Use cvmfs_config showconfig to get the effective parameters.
    #
    
    CVMFS_SERVER_URL=${CERNVM_SERVER_URL:="http://fermicloud130.fnal.gov/@org@"}
    CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/fnal.gov.pub
    
  4. Repository configurations
    [root@fermicloud048 ~]# cat /etc/cvmfs/config.d/ifcvmfs.fnal.gov.conf
    CVMFS_SERVER_URL=http://fermicloud130.fnal.gov/ifcvmfs/prod
    
  5. public key
    [root@fermicloud048 ~]# cat /etc/cvmfs/keys/fnal.gov.pub
    -----BEGIN PUBLIC KEY-----
    ABCDEjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAukBusmYyFW8KJxVMmeCj
    N7vcU1mERMpDhPTa5PgFROSViiwbUsbtpP9CvfxB/KU1gggdbtWOTZVTQqA3b+p8
    g5Vve3/rdnN5ZEquxeEfIG6iEZta9Zei5mZMeuK+DPdyjtvN1wP0982ppbZzKRBu
    BbzR4YdrwwWXXNZH65zZuUISDJB4my4XRoVclrN5aGVz4PjmIZFlOJ+ytKsMlegW
    SNDwZO9z/YtBFil/Ca8FJhRPFMKdvxK+ezgq+OQWAerVNX7fArMC+4Ya5pF3ASr6
    3mlvIsBpejCUBygV4N2pxIcPJu/ZDaikmVvdPTNOTZlIFMf4zIP/YHegQSJmOyVp
    HQID7890
    -----END PUBLIC KEY-----
    

Validation

  1. service cvmfs restart
  2. service cvmfs probe
  3. cvmfs_config checksetup
  4. cvmfs_config showconfig repository.fnal.gov
  5. mkdir /mnt/cvmfs
  6. mount -t cvmfs repository.fnal.gov /mnt/cvmfs
  7. service cvmfs restartautofs

NuComp Demo stuff

CVM-FS has been installed on Fermicloud VM's.

g minus 2 prototype
One pair of nodes have the g minus 2 experiment software release on a disk area that is part of the VM image.

  1. gm2 cvmfs server - fermicloud055
    [root@fermicloud055 ~]# service httpd status
    httpd (pid  11918) is running...
    [root@fermicloud055 ~]# service cvmfsd status
    CernVM-FS daemon runnning as pid 2052
    [root@fermicloud055 ~]# ls /cloud/images/lueking/gm2/app/
    cvmfs-test-dir1  external  g-2
    [root@fermicloud055 ~]# ls /cloud/images/lueking/gm2/app/external
    art            clhep    fftw      geant4      libxml2          scons  xerces_c
    boost          cmake    fhiclcpp  geant4data  messagefacility  setup
    cetbuildtools  cpp0x    gcc       heppdt      python           upd
    cetlib         cppunit  gccxml    libsigcpp   root             ups
    [root@fermicloud055 ~]# df
    Filesystem           1K-blocks      Used Available Use% Mounted on
    /dev/hda1              9912560   4265580   5135312  46% /
    tmpfs                  2021660         0   2021660   0% /dev/shm
    /dev/ram0                15863      1180     13864   8% /etc/cloud-security
    blue2.fnal.gov:/fermicloud/images
                         3221225472 1908551648 1312673824  60% /cloud/images
    

    Changes made to the "shadow tree" area are recorded in the change log. The "web catalog" area is synchronized, based on the change log, with the following:
    [root@fermicloud055 ~]# cvmfs-sync
    Locking down /cloud/images/lueking/gm2/app                 [  OK  ]
    Waiting for pending operations on /cloud/images/lueking/gm2[  OK  ]
    Waiting for pending messages in call buffer                [  OK  ]
    Stopping journal writer                                    [  OK  ]
    Attaching /var/www/webfs/gm2/app/pub/catalogs/.cvmfscatalog.working
    Attaching /var/www/webfs/gm2/app/pub/catalogs/cvmfs-test-dir1/.cvmfscatalog.work
    ing
    Parsing file system change log... 2 lines
    Post-processing file system change log...
    Step 1 - Deleting obsolete file and directory entries (0 entries):
    Step 2 - Inserting new directories and symlinks (0 entries):
    Step 3 - Updating touched directories (0 entries):
    Step 4 - Building file list (0 entries):
    Step 5 - Compressing and calculating content hashes using 4 threads (0 files):
    Step 6 - Updating file catalogs (0 files):
    Commit changes to catalogs...
    Creating catalog snapshot at /cloud/images/lueking/gm2/app
    Flushing file system change log                            [  OK  ]
    Resuming journal writer                                    [  OK  ]
    Open up /cloud/images/lueking/gm2/app                      [  OK  ]
    [root@fermicloud055 ~]#
    
  2. gm2 cvmfs client - fermicloud121

The client mount is performed as root, but any user has access to the mountpoint.

cvmfs2_debug -o 'fsname=cvmfs2,ro,nodev,grab_mountpoint,kernel_cache,auto_cache,uid=0,gid=0,cachedir=/var/cache/cvmfs2/gm2-prod.,quota_limit=61000,quota_threshold=30500,allow_other,entry_timeout=120,attr_timeout=120,timeout=120,negative_timeout=0,no_remote_lock,rebuild_cachedb,use_ino,proxies=http://fermicloud055.fnal.gov:80;http://fermicloud055.fnal.gov:80,timeout=5,timeout_direct=10,ro,nosuid,nodev,allow_other' /ifront/gm2/app 'http://fermicloud055.fnal.gov/gm2/app;http://fermicloud055.fnal.gov/gm2/app'

Logged in as a typical user on the client node.
-bash-3.2$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda1              9912560   1806284   7594608  20% /
tmpfs                  2021660         0   2021660   0% /dev/shm
/dev/ram0                15863      1175     13869   8% /etc/cloud-security
cvmfs2                62464000    245609  62218392   1% /ifront/gm2/app
blue2:/fermigrid-login/lueking
                     524288000 243086624 281201376  47% /home/lueking
-bash-3.2$ ls /ifront/gm2/app
cvmfs-test-dir1  external  g-2
-bash-3.2$ ls /ifront/gm2/app/external/
art            clhep    fftw      geant4      libxml2          scons  xerces_c
boost          cmake    fhiclcpp  geant4data  messagefacility  setup
cetbuildtools  cpp0x    gcc       heppdt      python           upd
cetlib         cppunit  gccxml    libsigcpp   root             ups
-bash-3.2$ . /ifront/gm2/app/external/setup
-bash-3.2$
-bash-3.2$ setup art v0_07_09 -q a2:debug
-bash-3.2$ gm2 --help
gm2 <-c <config-file>> <other-options> [<source-file>]+:
  -T [ --TFileName ] arg   File name for TFileService.
  -c [ --config ] arg      Configuration file.
  -e [ --estart ] arg      Event # of first event to process.
  -h [ --help ]            produce help message
  -n [ --nevts ] arg       Number of events to process.
  --nskip arg              Number of events to skip.
  -o [ --output ] arg      Event output stream file.
  -s [ --source ] arg      Source data file (multiple OK).
  -S [ --source-list ] arg file containing a list of source files to read, one
                           per line.
  --trace                  Activate tracing.
  --notrace                Deactivate tracing.
  --memcheck               Activate monitoring of memory use.
  --nomemcheck             Deactivate monitoring of memory use.

-bash-3.2$ which root
/ifront/gm2/app/external/root/v5_28_00a/Linux64bit+2.6-2.5-gcc45/bin/root
-bash-3.2$

Using /nusoft/app area for shadow tree.
The other pair of VM's demonstrate using /nusoft/app area, and ownership of the files as "nusoft" user.

  1. nusoft server - fermicloud130
    cvmfsd run as root. The "shadow tree" and web catalog are owned by "nusoft". Sync is done as user "nusoft"
-bash-3.2$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vda1              9912560   1527796   7873096  17% /
tmpfs                  2021660         0   2021660   0% /dev/shm
/dev/ram0                15863      1175     13869   8% /etc/cloud-security
blue3:/nusoft/app    536870912 164352864 372518048  31% /cvmfs/nusoft/app
blue3.fnal.gov:/nusoft/app
                     536870912 164352864 372518048  31% /nusoft/app
blue2:/fermigrid-login/lueking
                     524288000 243082720 281205280  47% /home/lueking

-bash-3.2$ ls -la /cvmfs/nusoft/app/ifcvmfs/prod
total 224
drwxr-xr-x 7 nusoft  nusoft 2048 Aug 31 08:57 .
drwxr-xr-x 3 nusoft  nusoft 2048 Aug 19 08:36 ..
-rw-r--r-- 1 nusoft  nusoft    0 Aug 31 08:52 .cvmfscatalog
drwxr-xr-x 2 lueking gm2    2048 Aug 16 14:24 gm2
drwxr-xr-x 2 nusoft  nusoft 2048 Aug 31 08:57 minerva
drwxrwxr-x 3 nova    p929   2048 Aug 19 08:52 nova
drwxr-xr-x 3 nusoft  nusoft 2048 Aug 31 08:22 nusoft
drwxr-xr-x 4 nusoft  nusoft 2048 Aug 31 08:15 test-sub-dir
-bash-3.2$
  1. nusoft client - fermicloud048
    As root...
    cvmfs2_debug -o 'fsname=cvmfs2,ro,nodev,grab_mountpoint,kernel_cache,auto_cache,uid=0,gid=0,cachedir=/var/cache/cvmfs2/ifcvmfs-prod.,quota_limit=61000,quota_threshold=30500,allow_other,entry_timeout=120,attr_timeout=120,timeout=120,negative_timeout=0,no_remote_lock,rebuild_cachedb,use_ino,proxies=http://fermicloud130.fnal.gov:80;http://fermicloud130.fnal.gov:80,timeout=5,timeout_direct=10,ro,nosuid,nodev,allow_other' /ifcvmfs/prod 'http://fermicloud130.fnal.gov/ifcvmfs/prod;http://fermicloud130.fnal.gov/ifcvmfs/prod'
    

Unpriveledged user has access to /ifcvmfs/prod area

-bash-3.2$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/vda1              9912560   1519608   7881284  17% /
tmpfs                  2021660         0   2021660   0% /dev/shm
/dev/ram0                15863      1175     13869   8% /etc/cloud-security
cvmfs2                62464000        22  62463978   1% /ifcvmfs/prod
-bash-3.2$
-bash-3.2$ ls -la /ifcvmfs/prod
total 8
drwxr-xr-x 1 root root 2048 Aug 31 08:52 .
drwxr-xr-x 1 root root 2048 Aug 31 08:22 nusoft
drwxr-xr-x 1 root root 2048 Aug 31 08:15 test-sub-dir

Notes and such below

Setting up a FermiCloud VM

Information about FermiCloud is found at http://www-fermicloud.fnal.gov/. Here is the description of the virtual machine set up for this test.

log into fcl002

cd OpenNebula
-bash-3.2$ cat .one-env
export ONE_AUTH=~/OpenNebula/.one-auth
-bash-3.2$ source .one-env
-bash-3.2$ onevm create cvmfstest.one
-bash-3.2$ onevm list
   ID     USER     NAME STAT CPU     MEM        HOSTNAME        TIME
  605 tlevshin tlevshin runn   0      4G          fcl009 27 23:54:24
  618  rsyoung rsyoung- runn   0      4G          fcl003 27 17:17:03
  622 tlevshin tlevshin runn   0      4G          fcl003 27 04:36:14
  626     neha neha-dat runn   0      2G          fcl008 27 03:14:30
  627     neha neha-dat runn   0      2G          fcl005 27 03:14:28
  628     neha neha-dat runn   0      2G          fcl004 27 03:14:26
  631 jdem-ope jdem-hea runn   0     16G          fcl005 27 01:56:36
  632 jdem-ope jdem-wor runn   0     16G          fcl004 27 01:56:25
  633 jdem-ope jdem-wor runn   0     16G          fcl008 27 01:56:18
  634 jdem-ope jdem-wor runn   0     16G          fcl009 27 01:56:15
  638 tlevshin tlevshin runn   0      4G          fcl003 24 01:44:03
  642 tlevshin tlevshin runn   0      4G          fcl004 23 21:07:43
  657 tlevshin   xrootd runn   0      2G          fcl006 22 23:00:18
  660 tlevshin   xrootd runn   0      2G          fcl006 22 22:27:08
  661 tlevshin   xrootd runn   0      2G          fcl005 22 22:27:03
  664     neha BestmanH runn   0      4G          fcl005 22 21:51:34
  667 tlevshin xrootdfs runn   0      4G          fcl006 22 03:28:25
  669 tlevshin   xrootd runn   0      2G          fcl007 22 03:08:38
  671 tlevshin     mock runn   0      4G          fcl009 22 00:11:38
  674 tlevshin tlevshin runn   0      4G          fcl003 21 17:57:52
  692  rsyoung   condor runn   0   1024M          fcl003 20 16:00:18
  693  rsyoung eco-vm-m runn   0   1024M          fcl006 20 15:42:49
  695     timm cloudsaz runn   1      4G          fcl010 20 04:06:04
  696     timm cloudsaz runn   0      4G          fcl010 20 04:05:56
  697     timm cloudlvs runn   1      2G          fcl010 20 03:51:28
  698     timm cloudgum runn   0      3G          fcl010 20 03:44:40
  699     timm cloudgum runn   1      3G          fcl010 20 03:44:32
  700     timm cloudmys runn   0      2G          fcl010 20 03:30:14
  701     timm cloudmys runn   0      2G          fcl010 20 03:30:11
  702    parag parag_te runn   0      4G          fcl008 20 02:53:12
  705     neha BestmanH runn   0      4G          fcl006 20 00:19:25
  706  dstrain rpmbuild runn   0      2G          fcl005 20 00:13:11
  717      dwd extenci0 runn   0      4G          fcl006 16 22:12:41
  718  rsyoung cloudgat runn   0      2G          fcl004 16 21:18:33
  726     lowe     lowe runn   0   1024M          fcl008 15 04:41:36
  732  rsyoung osgclien runn   0      4G          fcl007 14 01:16:37
  736     boyd samgfwd2 runn   0     10G          fcl007 10 04:48:10
  735  rsyoung     gums runn   0      4G          fcl007 12 23:05:18
  739    parag IFGridft runn   0      4G          fcl007 09 21:37:37
  743     timm test_lin runn   0      4G          fcl007 08 05:01:29
  746  dstrain  bestman runn   0      4G          fcl004 06 00:03:38
  747  dstrain  glidein runn   0      4G          fcl007 01 21:53:25
  748  dstrain  glidein runn   0      4G          fcl007 01 21:29:22
  754     lowe lowe-slf fail   0      0K          fcl007 00 00:06:51
  755  lueking  lueking pend   0      0K                 00 00:00:13
-bash-3.2$
-bash-3.2$ 

-bash-3.2$ onevm show 755
VIRTUAL MACHINE 755 INFORMATION
ID             : 755
NAME           : lueking
STATE          : ACTIVE
LCM_STATE      : RUNNING
START TIME     : 07/07 14:22:23
END TIME       : -
DEPLOY ID:     : one-755

VIRTUAL MACHINE MONITORING
NET_RX         : 0
USED MEMORY    : 0
USED CPU       : 0
NET_TX         : 0

VIRTUAL MACHINE TEMPLATE
CONTEXT=[
  FILES=/cloud/images/OpenNebula/templates/init.sh /home/lueking/OpenNebula/k5login,
  GATEWAY=131.225.154.1,
  IP_PUBLIC=131.225.154.153,
  NETMASK=255.255.254.0,
  NS=131.225.8.120,
  ROOT_PUBKEY=id_dsa.pub,
  TARGET=hdc,
  USERNAME=opennebula,
  USER_PUBKEY=id_dsa.pub ]

The /nusoft/data and /nusoft/app areas were added to /etc/fstab.

Installing CVMFS

[root@fermicloud070 ~]# cd /etc/pki/rpm-gpg/
[root@fermicloud070 rpm-gpg]# wget http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM
--2011-08-02 15:40:15--  http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM
Resolving cvmrepo.web.cern.ch... 137.138.139.146
Connecting to cvmrepo.web.cern.ch|137.138.139.146|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1734 (1.7K) [text/plain]
Saving to: `RPM-GPG-KEY-CernVM'

100%[======================================>] 1,734       --.-K/s   in 0s

2011-08-02 15:40:16 (3.57 MB/s) - `RPM-GPG-KEY-CernVM' saved [1734/1734]

[root@fermicloud070 rpm-gpg]# cd /etc/yum.repos.d/
[root@fermicloud070 yum.repos.d]# wget http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
--2011-08-02 15:41:44--  http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
Resolving cvmrepo.web.cern.ch... 137.138.139.146
Connecting to cvmrepo.web.cern.ch|137.138.139.146|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 175 [text/plain]
Saving to: `cernvm.repo'

100%[======================================>] 175         --.-K/s   in 0s

2011-08-02 15:41:45 (23.8 MB/s) - `cernvm.repo' saved [175/175]

[root@fermicloud070 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-0.2.68-1.x86_64.rpm
--2011-08-02 16:01:59--  https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-0.2.68-1.x86_64.rpm
Resolving cernvm.cern.ch... 137.138.234.20, 137.138.234.21
Connecting to cernvm.cern.ch|137.138.234.20|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 2848528 (2.7M) [application/x-rpm]
Saving to: `cvmfs-0.2.68-1.x86_64.rpm'

100%[======================================>] 2,848,528   1.15M/s   in 2.4s

2011-08-02 16:02:03 (1.15 MB/s) - `cvmfs-0.2.68-1.x86_64.rpm' saved [2848528/2848528]

[root@fermicloud070 ~]# rpm -ivh cvmfs-0.2.68-1.x86_64.rpm
warning: cvmfs-0.2.68-1.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
error: Failed dependencies:
        cvmfs-keys >= 1.1 is needed by cvmfs-0.2.68-1.x86_64
        fuse is needed by cvmfs-0.2.68-1.x86_64
[root@fermicloud070 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-keys-1.1-2.noarch.rpm
--2011-08-02 16:04:08--  https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-keys-1.1-2.noarch.rpm
Resolving cernvm.cern.ch... 137.138.234.20, 137.138.234.21
Connecting to cernvm.cern.ch|137.138.234.20|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 2942 (2.9K) [application/x-rpm]
Saving to: `cvmfs-keys-1.1-2.noarch.rpm'

100%[======================================>] 2,942       --.-K/s   in 0.002s

2011-08-02 16:04:09 (1.57 MB/s) - `cvmfs-keys-1.1-2.noarch.rpm' saved [2942/2942]

[root@fermicloud070 ~]# rpm -ivh cvmfs-keys-1.1-2.noarch.rpm
warning: cvmfs-keys-1.1-2.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
Preparing...                ########################################### [100%]
   1:cvmfs-keys             ########################################### [100%]
[root@fermicloud070 ~]# rpm -ivh cvmfs-0.2.68-1.x86_64.rpm
warning: cvmfs-0.2.68-1.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
error: Failed dependencies:
        fuse is needed by cvmfs-0.2.68-1.x86_64
[root@fermicloud070 ~]# yum install fuse
Loaded plugins: kernel-module
cernvm                                                   |  951 B     00:00
cernvm/primary                                           |  10 kB     00:00
cernvm                                                                    60/60
fermi-base                                               | 2.1 kB     00:00
fermi-base/primary_db                                    |  48 kB     00:00
fermi-security                                           | 1.9 kB     00:00
fermi-security/primary_db                                | 1.5 MB     00:00
sl-base                                                  | 2.1 kB     00:00
sl-base/primary_db                                       | 2.0 MB     00:00
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package fuse.x86_64 0:2.7.4-8.el5 set to be updated
--> Finished Dependency Resolution
Beginning Kernel Module Plugin
Finished Kernel Module Plugin

Dependencies Resolved

================================================================================
 Package        Arch             Version                Repository         Size
================================================================================
Installing:
 fuse           x86_64           2.7.4-8.el5            sl-base            83 k

Transaction Summary
================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)

Total download size: 83 k
Is this ok [y/N]: y
Downloading Packages:
fuse-2.7.4-8.el5.x86_64.rpm                              |  83 kB     00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : fuse                                                     1/1

Installed:
  fuse.x86_64 0:2.7.4-8.el5

Complete!
[root@fermicloud070 ~]# rpm -ivh cvmfs-0.2.68-1.x86_64.rpm
warning: cvmfs-0.2.68-1.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
Preparing...                ########################################### [100%]
   1:cvmfs                  ########################################### [100%]
[root@fermicloud070 ~]#                                          
  • server
    Installation instructions for installing the server are not readily available. I asked Jon Bakken about this and he confirmed this, and said they have installed this for CMS at Fermilab and Catlin can offer some assistance, rpm's etc. when he returns in 2 weeks (~Mid May). I found, from the RAL installation site () a link to a down load page (https://cernvm.cern.ch/project/trac/cernvm/downloads). Here I found the rpm cvmfs-server-0.2.68-1.x86_64.rpm. On the fermicloud vm I did the following:

[root@fermicloud004 ~]# yum install httpd

....all the output from that

[root@fermicloud020 ~]# yum install libgomp

....all the output from that

[root@fermicloud004 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-server-0.2.68-1.x86_64.rpm
--2011-05-04 14:52:55--  https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-server-0.2.68-1.x86_64.rpm
Resolving cernvm.cern.ch... 137.138.234.20, 137.138.234.21
Connecting to cernvm.cern.ch|137.138.234.20|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 3955653 (3.8M) [application/x-rpm]
Saving to: `cvmfs-server-0.2.68-1.x86_64.rpm'

100%[======================================>] 3,955,653   1.24M/s   in 3.0s

2011-05-04 14:52:59 (1.24 MB/s) - `cvmfs-server-0.2.68-1.x86_64.rpm' saved [3955653/3955653]

[root@fermicloud004 ~]# ls
anaconda-ks.cfg  cvmfs-server-0.2.68-1.x86_64.rpm  install.log.syslog
anaconda.log     install.log
[root@fermicloud004 ~]# rpm -ivh cvmfs-server-0.2.68-1.x86_64.rpm
warning: cvmfs-server-0.2.68-1.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
error: Failed dependencies:
        cvmfs-keys >= 1.1 is needed by cvmfs-server-0.2.68-1.x86_64
        httpd is needed by cvmfs-server-0.2.68-1.x86_64
        libgomp.so.1()(64bit) is needed by cvmfs-server-0.2.68-1.x86_64
        libgomp.so.1(GOMP_1.0)(64bit) is needed by cvmfs-server-0.2.68-1.x86_64
        libgomp.so.1(OMP_1.0)(64bit) is needed by cvmfs-server-0.2.68-1.x86_64
[root@fermicloud004 ~]# 
[root@fermicloud004 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-keys-1.1-2.noarch.rpm
--2011-05-04 15:20:27--  https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-keys-1.1-2.noarch.rpm
Resolving cernvm.cern.ch... 137.138.234.21, 137.138.234.20
Connecting to cernvm.cern.ch|137.138.234.21|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 2942 (2.9K) [application/x-rpm]
Saving to: `cvmfs-keys-1.1-2.noarch.rpm'

100%[======================================>] 2,942       --.-K/s   in 0s

2011-05-04 15:20:28 (55.0 MB/s) - `cvmfs-keys-1.1-2.noarch.rpm' saved [2942/2942]

[root@fermicloud004 ~]# rpm -ivh cvmfs-keys-1.1-2.noarch.rpm
warning: cvmfs-keys-1.1-2.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
Preparing...                ########################################### [100%]
   1:cvmfs-keys             ########################################### [100%]

----- new --- use this is in correct order

[root@fermicloud072 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-keys-1.1-2.noarch.rpm
--2011-07-21 13:36:32--  https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfs-keys-1.1-2.noarch.rpm
Resolving cernvm.cern.ch... 137.138.234.21, 137.138.234.20
Connecting to cernvm.cern.ch|137.138.234.21|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 2942 (2.9K) [application/x-rpm]
Saving to: `cvmfs-keys-1.1-2.noarch.rpm'

100%[======================================>] 2,942       --.-K/s   in 0.001s

2011-07-21 13:36:33 (3.44 MB/s) - `cvmfs-keys-1.1-2.noarch.rpm' saved [2942/2942]

[root@fermicloud072 ~]# rpm -ivh cvmfs-keys-1.1-2.noarch.rpm
warning: cvmfs-keys-1.1-2.noarch.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
Preparing...                ########################################### [100%]
   1:cvmfs-keys             ########################################### [100%]
[root@fermicloud072 ~]# rpm -ivh cvmfs-server-0.2.68-1.x86_64.rpm
warning: cvmfs-server-0.2.68-1.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
Preparing...                ########################################### [100%]
   1:cvmfs-server           ########################################### [100%]
[root@fermicloud072 ~]#                                                                                                                            

Need to get the httpd redirect conf.

[root@fermicloud072 ~]# scp root@fermicloud041.fnal.gov:/etc/httpd/conf.d/cvmfs.conf /etc/httpd/conf.d/.
cvmfs.conf                                    100% 1061     1.0KB/s   00:00

[root@fermicloud072 ~]# cat /etc/httpd/conf.d/cvmfs.conf
RewriteEngine on

#  - /opt/<experiment> is forced to be lower case
RewriteMap toLower int:tolower
RewriteLogLevel error
RewriteLog "/var/log/httpd/rewrite.log" 

# Lowering the case for repository names
RewriteRule ^/test/([A-Za-z0-9\-\.\_]+)/(.*)$ /test/${toLower:$1}/pub/catalogs/$2 [PT]

# Translation URL to real pathname
Alias /test /var/www/webfs/test

# Close access to non-pub directories
<DirectoryMatch /var/www/webfs/[A-Za-z0-9\-\.\_]+/(shadow|ctrl)>
    Order allow,deny
    Deny from all
</DirectoryMatch>

<Directory "/var/www/webfs">
    Options Indexes -MultiViews FollowSymLinks
    AllowOverride All
    Order allow,deny
    Allow from all

    IndexOptions +SuppressDescription +FoldersFirst +VersionSort +SuppressIcon

    EnableMMAP Off
    EnableSendFile Off

    AddType application/x-compress-cvmfs .cvmfschecksum

    FileETag INode MTime Size
    ExpiresActive On
    ExpiresDefault "access plus 60 minutes" 
    ExpiresByType text/html "access plus 60 minutes" 
    ExpiresByType application/x-compress-cvmfs "access" 
</Directory>
[root@fermicloud072 ~]#                     

Set up the client on "edge" machine

cd /etc/yum.repos.d
[root@edge yum.repos.d]# wget http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
--2011-05-09 11:28:47--  http://cvmrepo.web.cern.ch/cvmrepo/yum/cernvm.repo
Resolving cvmrepo.web.cern.ch... 137.138.139.146
Connecting to cvmrepo.web.cern.ch|137.138.139.146|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 175 [text/plain]
Saving to: `cernvm.repo'

100%[======================================>] 175         --.-K/s   in 0s

2011-05-09 11:28:48 (16.7 MB/s) - `cernvm.repo' saved [175/175]

[root@edge yum.repos.d]# ls
adobe.repo   fermi-addons.repo    fermi-security.repo  sl-srpms.repo
atrpms.repo  fermi-contrib.repo   fermi-testing.repo
cernvm.repo  fermi-fastbugs.repo  sl-debuginfo.repo
dag.repo     fermi.repo           sl.repo
[root@edge yum.repos.d]# cd /etc/pki/rpm-gpg
[root@edge rpm-gpg]# ls
RPM-GPG-KEY              RPM-GPG-KEY-jpolok
RPM-GPG-KEY-adobe-linux  RPM-GPG-KEY-redhat
RPM-GPG-KEY-atrpms       RPM-GPG-KEY-redhat-auxiliary
RPM-GPG-KEY-beta         RPM-GPG-KEY-redhat-beta
RPM-GPG-KEY-centos4      RPM-GPG-KEY-redhat-former
RPM-GPG-KEY-CentOS-5     RPM-GPG-KEY-redhat-release
RPM-GPG-KEY-cern         RPM-GPG-KEY-redhat-rhx
RPM-GPG-KEY-csieh        RPM-GPG-KEY-sl
RPM-GPG-KEY-dag          RPM-GPG-KEY-sl3
RPM-GPG-KEY-dawson       RPM-GPG-KEY-sl4
RPM-GPG-KEY-dries        RPM-GPG-KEY-sl5
RPM-GPG-KEY-fedora       RPM-GPG-KEY-sl6
RPM-GPG-KEY-fedora-test  RPM-GPG-KEY-wiesand
[root@edge rpm-gpg]# wget http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM
--2011-05-09 11:30:29--  http://cvmrepo.web.cern.ch/cvmrepo/yum/RPM-GPG-KEY-CernVM
Resolving cvmrepo.web.cern.ch... 137.138.139.146
Connecting to cvmrepo.web.cern.ch|137.138.139.146|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1734 (1.7K) [text/plain]
Saving to: `RPM-GPG-KEY-CernVM'

100%[======================================>] 1,734       --.-K/s   in 0s

2011-05-09 11:30:29 (150 MB/s) - `RPM-GPG-KEY-CernVM' saved [1734/1734]

[root@edge rpm-gpg]# cd
[root@edge ~]# yum install cvmfs-keys cvmfs cvmfs-init-scripts
Loaded plugins: kernel-module
cernvm                                                   |  951 B     00:00
cernvm/primary                                           | 8.3 kB     00:00
cernvm                                                                    47/47
Excluding Packages in global exclude list
Finished
Setting up Install Process
Package cvmfs-keys-1.1-2.noarch already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package cvmfs.x86_64 0:0.2.68-1 set to be updated
--> Processing Dependency: fuse for package: cvmfs
---> Package cvmfs-init-scripts.noarch 0:1.0.8-2 set to be updated
--> Running transaction check
---> Package fuse.x86_64 0:2.7.4-8.el5 set to be updated
--> Finished Dependency Resolution
Beginning Kernel Module Plugin
Finished Kernel Module Plugin

Dependencies Resolved

================================================================================
 Package                  Arch         Version              Repository     Size
================================================================================
Installing:
 cvmfs                    x86_64       0.2.68-1             cernvm        2.7 M
 cvmfs-init-scripts       noarch       1.0.8-2              cernvm        4.8 k
Installing for dependencies:
 fuse                     x86_64       2.7.4-8.el5          sl-base        83 k

Transaction Summary
================================================================================
Install      3 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 2.8 M
Is this ok [y/N]: y
Downloading Packages:
(1/3): cvmfs-init-scripts-1.0.8-2.noarch.rpm             | 4.8 kB     00:00
(2/3): fuse-2.7.4-8.el5.x86_64.rpm                       |  83 kB     00:00
(3/3): cvmfs-0.2.68-1.x86_64.rpm                         | 2.7 MB     00:01
--------------------------------------------------------------------------------
Total                                           1.0 MB/s | 2.8 MB     00:02
warning: rpmts_HdrFromFdno: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
cernvm/gpgkey                                            | 1.7 kB     00:00
Importing GPG key 0x8AE45CE7 "CernVM Administrator (cvmadmin) <cernvm.administrator@cern.ch>" from /etc/pki/rpm-gpg/RPM-GPG-KEY-CernVM
Is this ok [y/N]: y
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : fuse                                                     1/3
  Installing     : cvmfs                                                    2/3
  Installing     : cvmfs-init-scripts                                       3/3

Installed:
  cvmfs.x86_64 0:0.2.68-1          cvmfs-init-scripts.noarch 0:1.0.8-2

Dependency Installed:
  fuse.x86_64 0:2.7.4-8.el5

Complete!
[root@edge ~]#                                      
  • kernel module (from http://www.redirfs.org)
    note this is not needed to install this way, see the part on installing vmfsflt-kmod-common ...
[root@edge ~]# wget http://www.redirfs.org/packages/redirfs-0.10.tar.gz
--2011-05-10 10:38:12--  http://www.redirfs.org/packages/redirfs-0.10.tar.gz
Resolving www.redirfs.org... 147.229.12.248, 2001:67c:1220:80c::93e5:cf8
Connecting to www.redirfs.org|147.229.12.248|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 40924 (40K) [application/x-gzip]
Saving to: `redirfs-0.10.tar.gz'

100%[======================================>] 40,924       146K/s   in 0.3s

2011-05-10 10:38:13 (146 KB/s) - `redirfs-0.10.tar.gz' saved [40924/40924]

[root@edge ~]# tar -xvzf redirfs-0.10.tar.gz
redirfs-0.10/
redirfs-0.10/rfs_flt.c
redirfs-0.10/rfs.h
redirfs-0.10/redirfs.h
redirfs-0.10/rfs_info.c
redirfs-0.10/rfs_path.c
redirfs-0.10/CHANGELOG
redirfs-0.10/.svnignore
redirfs-0.10/rfs_inode.c
redirfs-0.10/rfs_ops.c
redirfs-0.10/README
redirfs-0.10/rfs_dentry.c
redirfs-0.10/rfs_sysfs.c
redirfs-0.10/rfs_dcache.c
redirfs-0.10/rfs_data.c
redirfs-0.10/TODO
redirfs-0.10/INSTALL
redirfs-0.10/COPYING
redirfs-0.10/rfs_file.c
redirfs-0.10/rfs.c
redirfs-0.10/rfs_chain.c
redirfs-0.10/Makefile
redirfs-0.10/rfs_root.c
[root@edge ~]# cd redirfs-0.10
[root@edge redirfs-0.10]# make -C /lib/modules/`uname -r`/build M=`pwd` modules
make: Entering directory `/usr/src/kernels/2.6.18-238.9.1.el5-x86_64'
  CC [M]  /root/redirfs-0.10/rfs_path.o
  CC [M]  /root/redirfs-0.10/rfs_root.o
  CC [M]  /root/redirfs-0.10/rfs_info.o
  CC [M]  /root/redirfs-0.10/rfs_file.o
  CC [M]  /root/redirfs-0.10/rfs_dentry.o
  CC [M]  /root/redirfs-0.10/rfs_inode.o
/root/redirfs-0.10/rfs_inode.c: In function ‘rfs_inode_del’:
/root/redirfs-0.10/rfs_inode.c:122: warning: assignment discards qualifiers from pointer target type
  CC [M]  /root/redirfs-0.10/rfs_dcache.o
  CC [M]  /root/redirfs-0.10/rfs_chain.o
  CC [M]  /root/redirfs-0.10/rfs_ops.o
  CC [M]  /root/redirfs-0.10/rfs_data.o
  CC [M]  /root/redirfs-0.10/rfs_flt.o
  CC [M]  /root/redirfs-0.10/rfs_sysfs.o
  CC [M]  /root/redirfs-0.10/rfs.o
  LD [M]  /root/redirfs-0.10/redirfs.o
  Building modules, stage 2.
  MODPOST
  CC      /root/redirfs-0.10/redirfs.mod.o
  LD [M]  /root/redirfs-0.10/redirfs.ko
make: Leaving directory `/usr/src/kernels/2.6.18-238.9.1.el5-x86_64'
[root@edge redirfs-0.10]# make -C /lib/modules/`uname -r`/build M=`pwd` modules_install
make: Entering directory `/usr/src/kernels/2.6.18-238.9.1.el5-x86_64'
  INSTALL /root/redirfs-0.10/redirfs.ko
  DEPMOD  2.6.18-238.9.1.el5
make: Leaving directory `/usr/src/kernels/2.6.18-238.9.1.el5-x86_64'
[root@edge redirfs-0.10]# depmod -a
[root@edge redirfs-0.10]# modprobe redirfs
[root@edge redirfs-0.10]#                                                     

vmfsflt-kmod-common is part of the kernel RPMs (cvmfsflt-0.2.68-1.x86_64.rpm, kmod-cvmfsflt-0.2.68-1.x86_64.rpm, and the corresponding redirfs kernel modules further down the page). As the modules depend on each other, the four modules have to be installed altogether.


[root@fermicloud020 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/redirfs-0.10-0.1.20101209svn671.x86_64.rpm
--2011-07-07 16:01:02--  https://cernvm.cern.ch/project/trac/downloads/cernvm/redirfs-0.10-0.1.20101209svn671.x86_64.rpm
Resolving cernvm.cern.ch... 137.138.234.21, 137.138.234.20
Connecting to cernvm.cern.ch|137.138.234.21|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 3701 (3.6K) [application/x-rpm]
Saving to: `redirfs-0.10-0.1.20101209svn671.x86_64.rpm'

100%[=====================================================>] 3,701       --.-K/s   in 0s

2011-07-07 16:01:03 (27.8 MB/s) - `redirfs-0.10-0.1.20101209svn671.x86_64.rpm' saved [3701/3701]

[root@fermicloud020 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/kmod-redirfs-0.10-0.1.20101209svn671.x86_64.rpm
--2011-07-07 16:01:23--  https://cernvm.cern.ch/project/trac/downloads/cernvm/kmod-redirfs-0.10-0.1.20101209svn671.x86_64.rpm
Resolving cernvm.cern.ch... 137.138.234.21, 137.138.234.20
Connecting to cernvm.cern.ch|137.138.234.21|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 392974 (384K) [application/x-rpm]
Saving to: `kmod-redirfs-0.10-0.1.20101209svn671.x86_64.rpm'

100%[=====================================================>] 392,974      239K/s   in 1.6s

2011-07-07 16:01:25 (239 KB/s) - `kmod-redirfs-0.10-0.1.20101209svn671.x86_64.rpm' saved [392974/392974]

[root@fermicloud020 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/kmod-redirfs-xen-0.10-0.1.20101209svn671.x86_64.rpm
--2011-07-07 16:01:43--  https://cernvm.cern.ch/project/trac/downloads/cernvm/kmod-redirfs-xen-0.10-0.1.20101209svn671.x86_64.rpm
Resolving cernvm.cern.ch... 137.138.234.21, 137.138.234.20
Connecting to cernvm.cern.ch|137.138.234.21|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 393262 (384K) [application/x-rpm]
Saving to: `kmod-redirfs-xen-0.10-0.1.20101209svn671.x86_64.rpm'

100%[=====================================================>] 393,262      287K/s   in 1.3s

2011-07-07 16:01:45 (287 KB/s) - `kmod-redirfs-xen-0.10-0.1.20101209svn671.x86_64.rpm' saved [393262/393262]

[root@fermicloud020 ~]# rpm -ivh cvmfsflt-0.2.68-1.x86_64.rpm kmod-cvmfsflt-0.2.68-1.x86_64.rpm redirfs-0.10-0.1.20101209svn671.x86_64.rpm kmod-redirfs-0.10-0.1.20101209svn671.x86_64.rpm
error: open of cvmfsflt-0.2.68-1.x86_64.rpm failed: No such file or directory
error: open of kmod-cvmfsflt-0.2.68-1.x86_64.rpm failed: No such file or directory
warning: redirfs-0.10-0.1.20101209svn671.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID 8ae45ce7
[root@fermicloud020 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/kmod-cvmfsflt-0.2.68-1.x86_64.rpm
--2011-07-07 16:03:23--  https://cernvm.cern.ch/project/trac/downloads/cernvm/kmod-cvmfsflt-0.2.68-1.x86_64.rpm
Resolving cernvm.cern.ch... 137.138.234.20, 137.138.234.21
Connecting to cernvm.cern.ch|137.138.234.20|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 124965 (122K) [application/x-rpm]
Saving to: `kmod-cvmfsflt-0.2.68-1.x86_64.rpm'

100%[=====================================================>] 124,965      113K/s   in 1.1s

2011-07-07 16:03:26 (113 KB/s) - `kmod-cvmfsflt-0.2.68-1.x86_64.rpm' saved [124965/124965]

[root@fermicloud020 ~]# wget --no-check-certificate https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfsflt-0.2.68-1.x86_64.rpm
--2011-07-07 16:04:44--  https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfsflt-0.2.68-1.x86_64.rpm
Resolving cernvm.cern.ch... 137.138.234.21, 137.138.234.20
Connecting to cernvm.cern.ch|137.138.234.21|:443... connected.
WARNING: cannot verify cernvm.cern.ch's certificate, issued by `/DC=ch/DC=cern/CN=CERN Trusted Certification Authority':
  Self-signed certificate encountered.
HTTP request sent, awaiting response... 200 OK
Length: 2424 (2.4K) [application/x-rpm]
Saving to: `cvmfsflt-0.2.68-1.x86_64.rpm'

100%[=====================================================>] 2,424       --.-K/s   in 0.001s

2011-07-07 16:04:45 (4.55 MB/s) - `cvmfsflt-0.2.68-1.x86_64.rpm' saved [2424/2424]

[root@edge ~]# rpm -ivh cvmfsflt-0.2.68-1.x86_64.rpm kmod-cvmfsflt-0.2.68-1.x86_64.rpm redirfs-0.10-0.1.20101209svn671.x86_64.rpm kmod-redirfs-0.10-0.1.20101209svn671.x86_64.rpm
Preparing...                ########################################### [100%]
   1:kmod-redirfs           ########################################### [ 25%]
   2:cvmfsflt               ########################################### [ 50%]
   3:kmod-cvmfsflt          ########################################### [ 75%]
   4:redirfs                ########################################### [100%]

Useful commands (client)

cvmfs_config showconfig ifront.fnal.gov
service cvmfs restart
service cvmfs probe 
cvmfs-talk 
cvmfs_config chksetup
mount -t cvmfs ifront.fnal.gov /mnt/cvmfs

Useful commands (server)

service cvmfsd restart
cvmfs-sync  #calls cvmfs_sync.bin w/ args from /etc/cvmfsdrc
cvmfs_sync.bin 
/etc/init.d/cvmfsd start
/etc/init.d/cvmfsd status
service cvmfsd restart 

Setting up catalog signing (I turned this off for testing)

[root@fermicloud004 ~]# openssl genrsa -out /tmp/cvmfs_test.key 2048
Generating RSA private key, 2048 bit long modulus
...............+++
...........................................................................................................................+++
e is 65537 (0x10001)
[root@fermicloud004 ~]# openssl req -new -subj "/C=US/ST=Illinois/L=Batavia/O=FNAL/OU=PH-SFT/CN=fermicloud004.fnal.gov" -key /tmp/cvmfs_test.key -out /tmp/cvmfs_test.csr
[root@fermicloud004 ~]# openssl x509 -req -days 365 -in /tmp/cvmfs_test.csr -signkey /tmp/cvmfs_test.key -out /tmp/cvmfs_test.crt
Signature ok
subject=/C=US/ST=Illinois/L=Batavia/O=FNAL/OU=PH-SFT/CN=fermicloud004.fnal.gov
Getting Private key
[root@fermicloud004 ~]# cvmfs_sign -c /tmp/cvmfs_test.crt -k /tmp/cvmfs_test.key -n 127.0.0.1 /var/www/html/catalogs/.cvmfspublished
Signing /var/www/html/catalogs/.cvmfscatalog
Failed to open /var/www/html/catalogs/.cvmfschecksum
[root@fermicloud004 ~]# ls /var/www
cgi-bin  cvmfs  error  html  icons  manual  usage
[root@fermicloud004 ~]# ls /var/www/html
cvmfs  pub
[root@fermicloud004 ~]# ls /var/www/html/cvmfs
catalogs  data  ifront.fnal.gov
[root@fermicloud004 ~]# cvmfs_sign -c /tmp/cvmfs_test.crt -k /tmp/cvmfs_test.key -n 127.0.0.1 /var/www/html/cvmfs/catalogs/.cvmfspublished
Signing /var/www/html/cvmfs/catalogs/.cvmfscatalog
[root@fermicloud004 ~]# ls /var/www/html/cvmfs/catalogs
data  ifront.fnal.gov
[root@fermicloud004 ~]# openssl x509 -fingerprint -sha1 -in /tmp/cvmfs_test.crt | grep "SHA1 Fingerprint" | sed 's/SHA1 Fingerprint=//' > /tmp/whitelist.test.unsigned
[root@fermicloud004 ~]# echo `date -u "+%Y%m%d%H%M%S"` > /tmp/whitelist.test.signed
[root@fermicloud004 ~]# echo "E`date -u --date='next month' "+%Y%m%d%H%M%S"`" >> /tmp/whitelist.test.signed
[root@fermicloud004 ~]# echo "N127.0.0.1" >> /tmp/whitelist.test.signed
[root@fermicloud004 ~]# cat /tmp/whitelist.test.unsigned >> /tmp/whitelist.test.signed
[root@fermicloud004 ~]# sha1=`openssl sha1 < /tmp/whitelist.test.signed | head -c40`
[root@fermicloud004 ~]# echo "--" >> /tmp/whitelist.test.signed
[root@fermicloud004 ~]# echo $sha1 >> /tmp/whitelist.test.signed
[root@fermicloud004 ~]# echo $sha1 | head -c 40 > /tmp/whitelist.test.sha1
[root@fermicloud004 ~]# openssl genrsa -out /tmp/cvmfs_master.key 2048
Generating RSA private key, 2048 bit long modulus
..............................................................+++
...........+++
e is 65537 (0x10001)
[root@fermicloud004 ~]# openssl rsa -in /tmp/cvmfs_master.key -pubout -out /tmp/cvmfs_master.pub
writing RSA key
[root@fermicloud004 ~]# openssl rsautl -inkey /tmp/cvmfs_master.key -sign -in /tmp/whitelist.test.sha1 -out /tmp/whitelist.test.signature
[root@fermicloud004 ~]# cat /tmp/whitelist.test.signature >> /tmp/whitelist.test.signed
[root@fermicloud004 ~]# cp /tmp/whitelist.test.signed /var/www/html/cvmfs/catalogs/.cvmfswhitelist
[root@fermicloud004 ~]# dd if=/dev/zero of=/tmp/cvmfs.faulty bs=1024 count=8
8+0 records in
8+0 records out
8192 bytes (8.2 kB) copied, 0.000456 seconds, 18.0 MB/s
[root@fermicloud004 ~]# service cvmfs restartclean
Shutting down CernVM-FS:                                   [  OK  ]
Starting CernVM-FS:                                        [  OK  ]
[root@fermicloud004 ~]#                                                
  • cvmfsdrc config file on server
[root@fermicloud020 cvmfsd]# cat /etc/cvmfsdrc
# The directory that shall be monitored
# and where you install and test your software
# Required.
SHADOW_DIR=/test/prod

# The repository storage.  Will be served by the webserver.
# Will be synchronized with SHADOW_DIR by cvmfs_sync.
# Required.
PUB_DIR=/var/www/webfs/test/prod/pub

# The file system change log of SHADOW_DIR.
# Used for synchronization.
# Required.
LOG_FILE=/var/log/cvmfsd/fschanges

# Number of threads for compression and hashing
# Optional
# CVMFS_SYNC_THREADS=1

# Nested catalog un subpaths of immutable directory trees
# are ignored for synchronization.
# Optional
# IMMUTABLES=/opt/zeppelin/old-release,/opt/zeppelin/very-old-release

# Whether to create CVMFS1 (GROW-FS) catalogs or not
# Optional, default=no
# COMPAT_CATALOG=yes

# Whether to sign catalogs during synchronization
# Optional, default=no
# SIGN_CATALOG=yes

# Whether to load only necessary catalog on synchronzation
# While this improve the scale to several thousand of catalogs
# it possibly breaks cross-catalog hard links
# Optional, default=no
# CVMFS_LAZYATTACH=yes

# Wether to maintain a list of dirty catalogs.  Useful
# for signing only changed catalogs.
# Optional.
# CVMFS_BOOKKEEPING=/var/log/cvmfsd/dirty_catalogs

# Sets the binding between whitelist and repository
# Optional.
# CVMFS_REPOSITORY_NAME=repository.domain
[root@fermicloud020 cvmfsd]#                      
  • syncing server with shadow directory
[root@fermicloud020 cvmfsd]# cvmfs-sync
Locking down /test/prod                                    [  OK  ]
Waiting for pending operations on /test/prod               [  OK  ]
Waiting for pending messages in call buffer                [  OK  ]
Stopping journal writer                                    [  OK  ]
Attaching /var/www/webfs/test/prod/pub/catalogs/.cvmfscatalog.working
creating root entry
Parsing file system change log... 3 lines
Post-processing file system change log...
Step 1 - Deleting obsolete file and directory entries (0 entries):
Step 2 - Inserting new directories and symlinks (0 entries):
Step 3 - Updating touched directories (0 entries):
Step 4 - Building file list (0 entries):
Step 5 - Compressing and calculating content hashes using 4 threads (0 files):
Step 6 - Updating file catalogs (0 files):
Commit changes to catalogs...
Flushing file system change log                            [  OK  ]
Resuming journal writer                                    [  OK  ]
Open up /test/prod                                         [  OK  ]
[root@fermicloud020 cvmfsd]# ls /test/prod
testme2.dat  testme.dat
[root@fermicloud020 cvmfsd]# vi /test/prod/testme2.dat
[root@fermicloud020 cvmfsd]# vi /test/prod/testme.dat
[root@fermicloud020 cvmfsd]# cvmfs-sync
Locking down /test/prod                                    [  OK  ]
Waiting for pending operations on /test/prod               [  OK  ]
Waiting for pending messages in call buffer                [  OK  ]
Stopping journal writer                                    [  OK  ]
Attaching /var/www/webfs/test/prod/pub/catalogs/.cvmfscatalog.working
Parsing file system change log... 42 lines
Post-processing file system change log...
Step 1 - Deleting obsolete file and directory entries (2 entries): Warning: /test/prod/testme.dat is not in the catalogs
.Warning: /test/prod/testme2.dat is not in the catalogs

Step 2 - Inserting new directories and symlinks (0 entries):
Step 3 - Updating touched directories (0 entries):
Step 4 - Building file list (2 entries): .
Step 5 - Compressing and calculating content hashes using 4 threads (2 files): .
Step 6 - Updating file catalogs (2 files): .
Commit changes to catalogs...
Creating catalog snapshot at /test/prod
Flushing file system change log                            [  OK  ]
Resuming journal writer                                    [  OK  ]
Open up /test/prod                                         [  OK  ]
[root@fermicloud020 cvmfsd]#                           
  • testing with debug mount on client
[root@edge cvmfs]# cvmfs2_debug -d -o 'fsname=test2,ro,nodev,grab_mountpoint,kernel_cache,auto_cache,uid=0,gid=0,cachedir=/scratch/lueking/cache/test2/prod.,quota_limit=61000,quota_threshold=30500,allow_other,entry_timeout=120,attr_timeout=120,timeout=120,negative_timeout=0,no_remote_lock,rebuild_cachedb,use_ino,proxies=http://fermicloud020.fnal.gov:80;http://fermicloud020.fnal.gov:80,timeout=5,timeout_direct=10,ro,nosuid,nodev,allow_other' /test/prod 'http://fermicloud020.fnal.gov/test/prod;http://fermicloud020.fnal.gov/test/prod'
( curl ) set host 0 to http://localhost    [07-08-2011 14:28:50 CDT]
( curl ) set host 0 to http://fermicloud020.fnal.gov/test/prod    [07-08-2011 14:28:50 CDT]
( curl ) set host 1 to http://fermicloud020.fnal.gov/test/prod    [07-08-2011 14:28:50 CDT]
( curl ) set proxy group 0 to http://fermicloud020.fnal.gov:80    [07-08-2011 14:28:50 CDT]
( curl ) set proxy group 1 to http://fermicloud020.fnal.gov:80    [07-08-2011 14:28:50 CDT]
( curl ) applying new lb group http://fermicloud020.fnal.gov:80    [07-08-2011 14:28:50 CDT]
( curl ) set lb member 0 to http://fermicloud020.fnal.gov:80    [07-08-2011 14:28:50 CDT]
( curl ) switching to forward proxy http://fermicloud020.fnal.gov:80 (number 0), 0 remaining in group    [07-08-2011 14:28:50 CDT]
( curl ) set timeout to 5 seconds    [07-08-2011 14:28:50 CDT]
Warning: current limits for number of open files are 1024/1024
CernVM-FS is likely to run out of file descriptors, set ulimit -n to at least 8192
CernVM-FS: using public key /etc/cvmfs/keys/cern.ch.pub
CernVM-FS: rebuilding lru cache database...
( lru ) re-building cache-database    [07-08-2011 14:28:50 CDT]
CernVM-FS: quota initialized, current size 0MB
( cvmfs ) searching for filesystem at http://fermicloud020.fnal.gov/test/prod;http://fermicloud020.fnal.gov/test/prod    [07-08-2011 14:28:50 CDT]
( cvmfs ) local checksum file is ./cvmfs.checksum.http---fermicloud020.fnal.gov-test-prod;http---fermicloud020.fnal.gov-test-prod    [07-08-2011 14:28:50 CDT]
( cvmfs ) unable to read local checksum    [07-08-2011 14:28:50 CDT]
( curl ) escaped http://fermicloud020.fnal.gov/test/prod/.cvmfspublished to http://fermicloud020.fnal.gov/test/prod/.cvmfspublished    [07-08-2011 14:28:50 CDT]
( cvmfs ) remote checksum is cacc5c6aba3a16495084cb2201c5444797b63c6b    [07-08-2011 14:28:50 CDT]
( cvmfs ) remote checksum is not signed    [07-08-2011 14:28:50 CDT]
( curl ) escaped http://fermicloud020.fnal.gov/test/prod/data/ca/cc5c6aba3a16495084cb2201c5444797b63c6bC to http://fermicloud020.fnal.gov/test/prod/data/ca/cc5c6aba3a16495084cb2201c5444797b63c6bC    [07-08-2011 14:28:50 CDT]
( lru ) pin into lru cacc5c6aba3a16495084cb2201c5444797b63c6b, path http://fermicloud020.fnal.gov/test/prod;http://fermicloud020.fnal.gov/test/prod    [07-08-2011 14:28:50 CDT]
( catalog ) Attaching ./cvmfs.catalog.gIUitZ as id 0    [07-08-2011 14:28:50 CDT]
( catalog ) Creating / Checking DB in ./cvmfs.catalog.gIUitZ    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: SELECT 0, hash, inode, size, mode, mtime, flags, name, symlink FROM catalog WHERE (parent_1 = :p_1) AND (parent_2 = :p_2);    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: SELECT 0, hash, inode, size, mode, mtime, flags, name, symlink FROM catalog WHERE (md5path_1 = :md5_1) AND (md5path_2 = :md5_2);    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: SELECT 0 FROM catalog WHERE (inode = :ino);    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: SELECT 0, a.hash, a.inode, a.size, a.mode, a.mtime, a.flags, a.name, a.symlink FROM catalog a, catalog b WHERE (b.md5path_1 = :md5_1) AND (b.md5path_2 = :md5_2) AND (a.md5path_1 = b.parent_1) AND (a.md5path_2 = b.parent_2);    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: SELECT sha1 FROM nested_catalogs WHERE path=:path;    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: INSERT INTO catalog (md5path_1, md5path_2, parent_1, parent_2, hash, inode, size, mode, mtime, flags, name, symlink) VALUES (:md5_1, :md5_2, :p_1, :p_2, :hash, :ino, :size, :mode, :mtime, :flags, :name, :symlink);    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: UPDATE catalog SET hash = :hash, size = :size, mode = :mode, mtime = :mtime, flags = :flags, name = :name, symlink = :symlink, inode = :inode WHERE (md5path_1 = :md5_1) AND (md5path_2 = :md5_2);    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: UPDATE catalog SET hash = :hash, size = :size, mode = coalesce(:mode, mode), mtime = :mtime WHERE inode = :ino;    [07-08-2011 14:28:50 CDT]
( catalog ) Prepared statement catalog 0: DELETE FROM catalog WHERE (md5path_1 = :md5_1) AND (md5path_2 = :md5_2);    [07-08-2011 14:28:50 CDT]
( cvmfs ) initial catalog load results in 0    [07-08-2011 14:28:50 CDT]
( talk ) socket created at ./cvmfs_io    [07-08-2011 14:28:50 CDT]
CernVM-FS: mounted cvmfs on /test/prod
CernVM-FS: linking to remote directoy http://fermicloud020.fnal.gov/test/prod;http://fermicloud020.fnal.gov/test/prod
FUSE library version: 2.8.4
nullpath_ok: 0
unique: 1, opcode: INIT (26), nodeid: 0, insize: 56
INIT: 7.10
flags=0x0000000b
max_readahead=0x00020000
   INIT: 7.12
   flags=0x00000010
   max_readahead=0x00020000
   max_write=0x00020000
   unique: 1, success, outsize: 40
( lru ) starting touch thread    [07-08-2011 14:28:51 CDT]
( lru ) starting insert thread    [07-08-2011 14:28:51 CDT]
( talk ) talk thread started    [07-08-2011 14:28:51 CDT]

[1]+  Stopped                 cvmfs2_debug -d -o 'fsname=test2,ro,nodev,grab_mountpoint,kernel_cache,auto_cache,uid=0,gid=0,cachedir=/scratch/lueking/cache/test2/prod.,quota_limit=61000,quota_threshold=30500,allow_other,entry_timeout=120,attr_timeout=120,timeout=120,negative_timeout=0,no_remote_lock,rebuild_cachedb,use_ino,proxies=http://fermicloud020.fnal.gov:80;http://fermicloud020.fnal.gov:80,timeout=5,timeout_direct=10,ro,nosuid,nodev,allow_other' /test/prod 'http://fermicloud020.fnal.gov/test/prod;http://fermicloud020.fnal.gov/test/prod'
[root@edge cvmfs]# bg
[1]+ cvmfs2_debug -d -o 'fsname=test2,ro,nodev,grab_mountpoint,kernel_cache,auto_cache,uid=0,gid=0,cachedir=/scratch/lueking/cache/test2/prod.,quota_limit=61000,quota_threshold=30500,allow_other,entry_timeout=120,attr_timeout=120,timeout=120,negative_timeout=0,no_remote_lock,rebuild_cachedb,use_ino,proxies=http://fermicloud020.fnal.gov:80;http://fermicloud020.fnal.gov:80,timeout=5,timeout_direct=10,ro,nosuid,nodev,allow_other' /test/prod 'http://fermicloud020.fnal.gov/test/prod;http://fermicloud020.fnal.gov/test/prod' &
[root@edge cvmfs]# ls /test/prod
unique: 2, opcode: GETATTR (3), nodeid: 1, insize: 56
getattr /
( cvmfs ) stat /    [07-08-2011 14:29:33 CDT]
( cvmfs ) current time 1310153373, deadline 1310156930    [07-08-2011 14:29:33 CDT]
( cvmfs ) catalog cache MISS (getattr)    [07-08-2011 14:29:33 CDT]
   unique: 2, success, outsize: 120
unique: 3, opcode: OPENDIR (27), nodeid: 1, insize: 48
   unique: 3, success, outsize: 32
unique: 4, opcode: READDIR (28), nodeid: 1, insize: 80
readdir[0] from 0
( cvmfs ) readdir /    [07-08-2011 14:29:33 CDT]
( cvmfs ) catalog cache HIT    [07-08-2011 14:29:33 CDT]
( cvmfs ) Found entry  in catalog 0, check if directory    [07-08-2011 14:29:33 CDT]
   unique: 4, success, outsize: 128
unique: 5, opcode: LOOKUP (1), nodeid: 1, insize: 51
LOOKUP /testme.dat
getattr /testme.dat
( cvmfs ) stat /testme.dat    [07-08-2011 14:29:33 CDT]
( cvmfs ) current time 1310153373, deadline 1310156930    [07-08-2011 14:29:33 CDT]
( cvmfs ) catalog cache MISS (getattr)    [07-08-2011 14:29:33 CDT]
   NODEID: 2
   unique: 5, success, outsize: 144
unique: 6, opcode: LOOKUP (1), nodeid: 1, insize: 52
LOOKUP /testme2.dat
getattr /testme2.dat
( cvmfs ) stat /testme2.dat    [07-08-2011 14:29:33 CDT]
( cvmfs ) current time 1310153373, deadline 1310156930    [07-08-2011 14:29:33 CDT]
( cvmfs ) catalog cache MISS (getattr)    [07-08-2011 14:29:33 CDT]
   NODEID: 3
   unique: 6, success, outsize: 144
unique: 7, opcode: READDIR (28), nodeid: 1, insize: 80
   unique: 7, success, outsize: 16
testme2.dat  testme.dat
[root@edge cvmfs]# unique: 8, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
   unique: 8, success, outsize: 16
l /test/prod/testme /test/prod/testme.dat
unique: 9, opcode: ACCESS (34), nodeid: 1, insize: 48
   unique: 9, error: -38 (Function not implemented), outsize: 16
-bash: /test/prod/testme.dat: Permission denied
[root@edge cvmfs]# cat /test/prod/testme.dat
unique: 10, opcode: OPEN (14), nodeid: 2, insize: 48
open flags: 0x8000 /testme.dat
( cvmfs ) catalog cache HIT    [07-08-2011 14:30:34 CDT]
( cache ) miss ./06/c9eaf61acb00bf8cf112238953ecf66dbd73af (-1)    [07-08-2011 14:30:34 CDT]
( cache ) miss ./06/c9eaf61acb00bf8cf112238953ecf66dbd73af (-1)    [07-08-2011 14:30:34 CDT]
( cache ) loading /testme.dat    [07-08-2011 14:30:34 CDT]
( cache ) curl fetches /data/06/c9eaf61acb00bf8cf112238953ecf66dbd73af    [07-08-2011 14:30:34 CDT]
( cache ) begin ./txn/f61acb00bf8cf112238953ecf66dbd73afPjjMZl    [07-08-2011 14:30:34 CDT]
( curl ) escaped http://fermicloud020.fnal.gov/test/prod/data/06/c9eaf61acb00bf8cf112238953ecf66dbd73af to http://fermicloud020.fnal.gov/test/prod/data/06/c9eaf61acb00bf8cf112238953ecf66dbd73af    [07-08-2011 14:30:34 CDT]
( cache ) curl finished downloading of /data/06/c9eaf61acb00bf8cf112238953ecf66dbd73af, checksum: 06c9eaf61acb00bf8cf112238953ecf66dbd73af    [07-08-2011 14:30:34 CDT]
( cache ) trying to commit    [07-08-2011 14:30:34 CDT]
( cache ) commit ./06/c9eaf61acb00bf8cf112238953ecf66dbd73af ./txn/f61acb00bf8cf112238953ecf66dbd73afPjjMZl    [07-08-2011 14:30:34 CDT]
( lru ) insert into lru 06c9eaf61acb00bf8cf112238953ecf66dbd73af, path /testme.dat    [07-08-2011 14:30:34 CDT]
   open[19] flags: 0x8000 /testme.dat
   unique: 10, success, outsize: 32
unique: 11, opcode: READ (15), nodeid: 2, insize: 80
read[19] 4096 bytes from 0 flags: 0x8000
   read[19] 44 bytes from 0
   unique: 11, success, outsize: 60
I'm feeling much better
1
2
3
4
5
6
7
8
9
0
unique: 12, opcode: FLUSH (25), nodeid: 2, insize: 64
   unique: 12, error: -38 (Function not implemented), outsize: 16
[root@edge cvmfs]# unique: 13, opcode: RELEASE (18), nodeid: 2, insize: 64
release[19] flags: 0x8000
( cvmfs ) closeing file number 19    [07-08-2011 14:30:34 CDT]
   unique: 13, success, outsize: 16
( lru ) insert thread, got sha1 06c9eaf61acb00bf8cf112238953ecf66dbd73af    [07-08-2011 14:30:34 CDT]

[root@edge cvmfs]# ls /test/prod
unique: 14, opcode: GETATTR (3), nodeid: 1, insize: 56
getattr /
( cvmfs ) stat /    [07-08-2011 14:31:10 CDT]
( cvmfs ) current time 1310153470, deadline 1310156930    [07-08-2011 14:31:10 CDT]
( cvmfs ) catalog cache HIT (getattr)    [07-08-2011 14:31:10 CDT]
   unique: 14, success, outsize: 120
unique: 15, opcode: OPENDIR (27), nodeid: 1, insize: 48
   unique: 15, success, outsize: 32
unique: 16, opcode: READDIR (28), nodeid: 1, insize: 80
readdir[0] from 0
( cvmfs ) readdir /    [07-08-2011 14:31:10 CDT]
( cvmfs ) catalog cache HIT    [07-08-2011 14:31:10 CDT]
( cvmfs ) Found entry  in catalog 0, check if directory    [07-08-2011 14:31:10 CDT]
   unique: 16, success, outsize: 128
unique: 17, opcode: GETATTR (3), nodeid: 2, insize: 56
getattr /testme.dat
( cvmfs ) stat /testme.dat    [07-08-2011 14:31:10 CDT]
( cvmfs ) current time 1310153470, deadline 1310156930    [07-08-2011 14:31:10 CDT]
( cvmfs ) catalog cache HIT (getattr)    [07-08-2011 14:31:10 CDT]
   unique: 17, success, outsize: 120
unique: 18, opcode: READDIR (28), nodeid: 1, insize: 80
   unique: 18, success, outsize: 16
testme2.dat  testme.dat
unique: 19, opcode: RELEASEDIR (29), nodeid: 1, insize: 64
   unique: 19, success, outsize: 16
[root@edge cvmfs]# cat /test/prod/testme2.dat
unique: 20, opcode: LOOKUP (1), nodeid: 1, insize: 52
LOOKUP /testme2.dat
getattr /testme2.dat
( cvmfs ) stat /testme2.dat    [07-08-2011 14:31:34 CDT]
( cvmfs ) current time 1310153494, deadline 1310156930    [07-08-2011 14:31:34 CDT]
( cvmfs ) catalog cache HIT (getattr)    [07-08-2011 14:31:34 CDT]
   NODEID: 3
   unique: 20, success, outsize: 144
unique: 21, opcode: OPEN (14), nodeid: 3, insize: 48
open flags: 0x8000 /testme2.dat
( cvmfs ) catalog cache HIT    [07-08-2011 14:31:34 CDT]
( cache ) miss ./b5/bd0307088474cfb005c35f838da64efb8098e1 (-1)    [07-08-2011 14:31:34 CDT]
( cache ) miss ./b5/bd0307088474cfb005c35f838da64efb8098e1 (-1)    [07-08-2011 14:31:34 CDT]
( cache ) loading /testme2.dat    [07-08-2011 14:31:34 CDT]
( cache ) curl fetches /data/b5/bd0307088474cfb005c35f838da64efb8098e1    [07-08-2011 14:31:34 CDT]
( cache ) begin ./txn/07088474cfb005c35f838da64efb8098e1RJXbOy    [07-08-2011 14:31:34 CDT]
( curl ) escaped http://fermicloud020.fnal.gov/test/prod/data/b5/bd0307088474cfb005c35f838da64efb8098e1 to http://fermicloud020.fnal.gov/test/prod/data/b5/bd0307088474cfb005c35f838da64efb8098e1    [07-08-2011 14:31:34 CDT]
( cache ) curl finished downloading of /data/b5/bd0307088474cfb005c35f838da64efb8098e1, checksum: b5bd0307088474cfb005c35f838da64efb8098e1    [07-08-2011 14:31:34 CDT]
( cache ) trying to commit    [07-08-2011 14:31:34 CDT]
( cache ) commit ./b5/bd0307088474cfb005c35f838da64efb8098e1 ./txn/07088474cfb005c35f838da64efb8098e1RJXbOy    [07-08-2011 14:31:34 CDT]
( lru ) insert into lru b5bd0307088474cfb005c35f838da64efb8098e1, path /testme2.dat    [07-08-2011 14:31:34 CDT]
   open[19] flags: 0x8000 /testme2.dat
   unique: 21, success, outsize: 32
( lru ) insert thread, got sha1 b5bd0307088474cfb005c35f838da64efb8098e1    [07-08-2011 14:31:34 CDT]
unique: 22, opcode: READ (15), nodeid: 3, insize: 80
read[19] 4096 bytes from 0 flags: 0x8000
   read[19] 22 bytes from 0
   unique: 22, success, outsize: 38
help, I need sombody!
unique: 23, opcode: RELEASE (18), nodeid: 3, insize: 64
release[19] flags: 0x8000
( cvmfs ) closeing file number 19    [07-08-2011 14:31:34 CDT]
   unique: 23, success, outsize: 16
[root@edge cvmfs]#          

Note on starting gm2 server on fermicloud node

Once the installation is complete, do the following (This is my superstition, need to try out some more)

#From https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfstech-0.2.70-1.pdf page 29
#modprobe redirfs
#modprobe cvmfsflt
  if [ ! −c / dev / cvmfs ] ; then
     major =‘grep cvmfs /proc/devices | awk ’{print $1}’‘
     mknod /dev/cvmfs c $major 0
  fi
echo −n "a:i:$MY_FILTER_PATH" > /sys/fs/redirfs/filters/cvmfsflt/paths

Then, start the server

[root@fermicloud055 conf.d]# service cvmfsd start
Looking for /cloud/images/lueking/gm2                      [  OK  ]
Creating file system journal in /var/log/cvmfsd/fschanges  [  OK  ]
Loading redirfs kernel module                              [  OK  ]
Loading cvmfsflt kernel module                             [  OK  ]
Linking to character devive                                [  OK  ]
Setting filter path to /cloud/images/lueking/gm2           [  OK  ]
Starting journal writer                                    [  OK  ]
Starting CernVM-FS daemon                                  [  OK  ]

Add the files to the shadow area (remove any tar files if that is where the files were unpacked from)
[root@fermicloud055 conf.d]# cd /cloud/images/lueking/gm2
[root@fermicloud055 gm2]# ls
art            cetlib  cpp0x    fhiclcpp  geant4     libxml2          root   upd
boost          clhep   cppunit  gcc       heppdt     messagefacility  scons  ups
cetbuildtools  cmake   fftw     gccxml    libsigcpp  python           setup  xerces_c

put in the locations of nested catalog databases
 
[root@fermicloud055 gm2]# touch .cvmfscatalog

Then do the sync

[root@fermicloud055 gm2]# cvmfs-sync
Locking down /cloud/images/lueking/gm2                     [  OK  ]
Waiting for pending operations on /cloud/images/lueking/gm2[  OK  ]
Waiting for pending messages in call buffer                [  OK  ]
Stopping journal writer                                    [  OK  ]
Attaching /cloud/images/lueking/www/webfs/gm2/pub/catalogs/.cvmfscatalog.working

(walks through the sync process)

Resuming journal writer                                    [  OK  ]
Open up /cloud/images/lueking/gm2                          [  OK  ]

This happened when I had shadow and ebfs on same file system

[root@fermicloud055 gm2]# cvmfs-sync
Locking down /cloud/images/lueking/gm2                     [  OK  ]
Waiting for pending operations on /cloud/images/lueking/gm2[  OK  ]
Waiting for pending messages in call buffer                [  OK  ]
Stopping journal writer                                    [  OK  ]
Attaching /cloud/images/lueking/www/webfs/gm2/pub/catalogs/.cvmfscatalog.working
could not init root catalog
could not initialize catalog store
Resuming journal writer                                    [  OK  ]
Open up /cloud/images/lueking/gm2                          [  OK  ]
[root@fermicloud055 gm2]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             9.5G  1.5G  7.6G  17% /
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/ram0              16M  1.2M   14M   8% /etc/cloud-security
blue2.fnal.gov:/fermicloud/images
                      3.0T  1.6T  1.5T  51% /cloud/images
[root@fermicloud055 gm2]#                                          

Successful when I split them

/var/www/webfs/gm2/app/pub  is the web cache
/cloud/images/lueking/gm2/app is the shadow directory

The tar'd files art... and muon... worked ok. Encountered problem with the sync the externals file. Reported as follows:


lots and lots of these...

Warning: unsupported file type in line 694510
Warning: unsupported file type in line 694513
Warning: unsupported file type in line 694516
Warning: unsupported file type in line 694519
Warning: unsupported file type in line 694522
Warning: unsupported file type in line 694525
Warning: unsupported file type in line 694528
Warning: unsupported file type in line 694531
Warning: unsupported file type in line 694534
Warning: unsupported file type in line 694537
695194 lines
Post-processing file system change log...
Step 1 - Deleting obsolete file and directory entries (3 entries): Warning: /cloud/images/lueking/gm2/app/.updfiles/updconfig is not in the catalogs
.Warning: /cloud/images/lueking/gm2/app/.updfiles/updusr.pm is not in the catalogs
Warning: /cloud/images/lueking/gm2/app/.upsfiles/dbconfig is not in the catalogs

Step 2 - Inserting new directories and symlinks (11871 entries): ............
Step 3 - Updating touched directories (2 entries): Warning: directory /cloud/images/lueking/gm2/app/.updfiles was not in catalogs
.Warning: directory /cloud/images/lueking/gm2/app/.upsfiles was not in catalogs

Step 4 - Building file list (161598 entries): Warning: dangling file entry /cloud/images/lueking/gm2/app/.updfiles/updconfig
.Warning: dangling file entry /cloud/images/lueking/gm2/app/.updfiles/updusr.pm
Warning: dangling file entry /cloud/images/lueking/gm2/app/.upsfiles/dbconfig
.................................................................................................................................................................
Step 5 - Compressing and calculating content hashes using 4 threads (161595 files): ......................................................................................................................
Step 6 - Updating file catalogs (161595 files): ..................................................................................................................................................................
Commit changes to catalogs...
Creating catalog snapshot at /cloud/images/lueking/gm2/app
Flushing file system change log                            [  OK  ]
Resuming journal writer                                    [  OK  ]
Open up /cloud/images/lueking/gm2/app                      [  OK  ]
[root@fermicloud055 app]#                

  • Testing with /nusoft/app as shadow tree area
[root@fermicloud130 ~]# mount -t nfs blue3:/nusoft/app /cvmfs/nusoft/app

Change the /etc/cvmfsdrc  shadow directory to /cvmfs/nusoft/app

Flush the change log buffer

cvmfsd_ctrl flush

remove change logs 

rm /var/log/cvmfsd/*

h2. Installing NOvA software for repository

<pre>
(as root, stop the cvmfsd)
wget http://oink.fnal.gov/distro/art/art_externals-0.2.6-slf5-x86_64-gcc45.tar.gz
wget http://oink.fnal.gov/distro/art/art_externals-0.2.6-source.tar.gz
wget http://oink.fnal.gov/distro/art/art_suite-0.07.04-slf5-x86_64-nova.tar.gz
wget http://oink.fnal.gov/distro/art/art_suite-0.07.04-source.tar.gz
wget http://oink.fnal.gov/distro/art/nova_extras-0.2.5-noarch.tar.gz
wget http://oink.fnal.gov/distro/art/nova_extras-0.2.5-slf5-x86_64-gcc45.tar.gz
wget http://oink.fnal.gov/distro/art/nova_extras-0.2.5-source.tar.gz
(as root, start the cvmfsd)
tar -zxf art_externals-0.2.6-slf5-x86_64-gcc45.tar.gz
tar -zxf art_externals-0.2.6-source.tar.gz
tar -zxf art_suite-0.07.04-slf5-x86_64-nova.tar.gz
tar -zxf art_suite-0.07.04-source.tar.gz
tar -zxf nova_extras-0.2.5-noarch.tar.gz
tar -zxf nova_extras-0.2.5-slf5-x86_64-gcc45.tar.gz
tar -zxf nova_extras-0.2.5-source.tar.gz
(as root, stop the cvmfsd)
-bash-3.2$ rm *.tar.gz
(as root, start the cvmfsd)
-bash-3.2$ cvmfs-sync
-bash-3.2$ wget https://cdcvs.fnal.gov/redmine/projects/novaart/repository/raw/SRT_NOVA/scripts/nova_srt_bootstrap  
-bash-3.2$ chmod +x nova_srt_bootstrap
-bash-3.2$ mkdir -p novaart/novasoft
-bash-3.2$ ./nova_srt_bootstrap /cvmfs/nusoft/app/ifcvmfs/prod/nova/novaart
-bash-3.2$ source /cvmfs/nusoft/app/ifcvmfs/prod/nova/novaart/novasoft/srt/srt.sh
-bash-3.2$ wget https://cdcvs.fnal.gov/redmine/projects/novaart/repository/raw/SRT_NOVA/scripts/update-release  
-bash-3.2$ chmod +x update-release
-bash-3.2$ export CVSROOT=':pserver:anonymous@cdcvs.fnal.gov:/cvs/projects/novaart'
-bash-3.2$ ./update-release -rel S11.10.08
... a lot of these ...
cvs -d :pserver:anonymous@cdcvs.fnal.gov:/cvs/nova checkout -P-r s11-10-08 RawFileParser
cvs checkout: warning: failed to open /nusoft/app/home/nusoft/.cvspass for reading: Read-only file system
cvs checkout: warning: cannot open history file `/cvs/nova/CVSROOT/history' for write: Permission denied

</pre>

<pre>
-bash-3.2$ . setup_novasoft_nusoft_ifcvmfs.sh
ERROR: Found no match for product 'root'
ERROR: Action parsing failed on "unsetuprequired()" 
ERROR: Found no match for product 'root'
ERROR: Action parsing failed on "setupRequired(root      v5_28_00a -q nova)" 
ERROR: Found no match for product 'root'
ERROR: Action parsing failed on "setupRequired(root      v5_28_00a -q nova)" 
INFORMATIONAL: Product 'genie' (with qualifiers 'nova'), has no v3249 version (or may not exist)
INFORMATIONAL: Product 'geant4' (with qualifiers 'debug:gcc45'), has no v4_9_4_p01 version (or may not exist)
INFORMATIONAL: Product 'cry' (with qualifiers 'gcc45'), has no v1_5 version (or may not exist)
INFORMATIONAL: Product 'cstxsd' (with qualifiers ''), has no v3_3_0 version (or may not exist)
INFORMATIONAL: Product 'postgresql' (with qualifiers 'gcc45'), has no v8_4_7 version (or may not exist)
INFORMATIONAL: Product 'mysql_client' (with qualifiers 'gcc45'), has no v5_1_56 version (or may not exist)
INFORMATIONAL: Product 'xerces_c' (with qualifiers 'gcc45'), has no v3_1_1 version (or may not exist)
INFORMATIONAL: Product 'NumiBeamDB' (with qualifiers 'R52800aGCC45'), has no v0_1_1 version (or may not exist)
INFORMATIONAL: Product 'totalview' (with qualifiers ''), has no v8_9_0a version (or may not exist)
-bash-3.2$ which gcc
/cvmfs/nusoft/app/ifcvmfs/prod/nova/externals/gcc/v4_5_1/Linux64bit+2.6-2.5/bin/gcc

</pre>