Project

General

Profile

AFSRET {{last_modified}} MIGRATION NOTES

2013-08-22 fsus-minos02 unmounted code

We had overlooked the unmount of the RETIRED code volumes.

List of mounts

AL=/minos/data/users/kreymer/log/afs
AD=/afs/.fnal.gov/files/code/e875/RETIRED
cd ${AD}

for DIR in `ls` ; do 
    VOLU=`fs listquota ${DIR} | tail -1 | cut -f 1 -d ' '`
    HOST=`fs whereis ${DIR} | cut -f 6 -d ' '`
    printf "${DIR}\t${VOLU}\t${HOST}\n" 
done > ${AL}/CODERETIRED.mounts

Archive of data to /minos/app

    minsoft@minos27

AD=/afs/.fnal.gov/files/code/e875/RETIRED

cd ${AD}

for DIR in `ls` ; do
    fs listquota ${DIR}
    cp -ax  ${DIR} /minos/app/afs/${DIR}
    du -sm  ${DIR} /minos/app/afs/${DIR}
    date ; echo
done  2>&1 | tee /var/tmp/CODERETIRED.archive

Plan for rmmount

AL=/minos/data/users/kreymer/log/afs
AD=/afs/.fnal.gov/files/code/e875/RETIRED

cat ${AL}/CODERETIRED.mounts | while read MOUNT ; do
   DIR=`printf "${MOUNT}" | cut -f 1`
   echo fs rmmount ${AD}/${DIR}
        fs rmmount ${AD}/${DIR}
done

date

volrelease on flxi02

Plan for remount of RETIRED code directories in case it is needed :

AL=/minos/data/users/kreymer/log/afs
AD=/afs/.fnal.gov/files/code/e875/RETIRING

cat ${AL}/CODERETIRED.mounts | while read MOUNT ; do
   DIR=`printf "${MOUNT}" | cut -f 1`
   VOL=`printf "${MOUNT}" | cut -f 2`
   echo fs mkmount -dir ${AD}/${DIR} -vol ${VOL}
        fs mkmount -dir ${AD}/${DIR} -vol ${VOL}
done

date

volrelease on flxi02

2013-08-20 fsus-minos02 unmounted

Executed the rmmount and volrelease as documented below

Tue Aug 20 09:11:42 CDT 2013
Tue Aug 20 09:15:03 CDT 2013

2013-08-16 fsus-minos02 retired
Posted to RITM0061737 :

You may procede with the fsus-minos02 maintenance at your convenience.

Minos is no longer using the fsus-minos02 file systems.
The files have been archived to /minos/data/afs

These directories are temporarily mounted for reference on
/afs/fnal.gov/files/data/minos/RETIRING

We plan to dismount the RETIRING directories Tuesday Aug 27 2013.

Thanks for your patience !

Plan for dismount of RETIRING directories Aug 20 :

AL=/minos/data/users/kreymer/log/afs
AG=/afs/.fnal.gov/files/data/minos/RETIRING

cat ${AL}/RETIRING.mounts | while read MOUNT ; do
   DIR=`printf "${MOUNT}" | cut -f 1`
   echo fs rmmount ${AG}/${DIR}
        fs rmmount ${AG}/${DIR}
done

date

volrelease on flxi02

Plan for remount of RETIRING directories in case it is needed :

AL=/minos/data/users/kreymer/log/afs
AG=/afs/.fnal.gov/files/data/minos/RETIRING

cat ${AL}/RETIRING.mounts | while read MOUNT ; do
   DIR=`printf "${MOUNT}" | cut -f 1`
   VOL=`printf "${MOUNT}" | cut -f 2`
   echo fs mkmount -dir ${AG}/${DIR} -vol ${VOL}
        fs mkmount -dir ${AG}/${DIR} -vol ${VOL}
done

date

volrelease on flxi02

While waiting for Aug 20, removed the emptied directories from 'retired'

AL=/minos/data/users/kreymer/log/afs

for DIR in `ls` ; do 
    VOLU=`fs listquota ${DIR} | tail -1 | cut -f 1 -d ' '`
    HOST=`fs whereis ${DIR} | cut -f 6 -d ' '`
    printf "${DIR}\t${VOLU}\t${HOST}\n" 
done > ${AL}/retired.mounts

cat ${AL}/retired.mounts | while read MOUNT ; do
   DIR=`printf "${MOUNT}" | cut -f 1`
   echo fs rmmount ${AR}/${DIR}
        fs rmmount ${AR}/${DIR}
done

date
  Fri Aug 16 12:11:05 CDT 2013

2013-08-15/16 fsus-minos02 final

data2 remounted as code/e875/data to avoid data/minos/data clutter
d210 copied DriftWeb to /minos/data/afs/d210/DriftWeb
Calibration removed DriftWeb and FD links, archived FD
log_data farm_backup.minospro was changed to write /minos/app/minospro/backup

This leaves only log_data/bluwatch to migrate.

log_data is served by fsus02, so does not need to migrate.
Remounted it on /afs/fnal.gov/files/data/minos.

Minos is no longer using the fsus-minos02 file systems.
The files have all been archived to /minos/data/afs

The directories are temporarily mounted for reference on
/afs/fnal.gov/files/data/minos/RETIRING

I plan to dismount the RETIRING directories Tuesday Aug 27.

2013-08-11/12/13 fsus-minos02 final cleanup
DATA CLEANUP

Now that the code areas are ready for shutdown,
reviewing the state of data migration under /afs/.fnal.gov/files/data/minos
This was left not-quite-finished.

The data/minos area contains mostly links to /minos/data except :

Directory Size Content
data2 0 empty will remount this as data/minos
RETIRING
calibration
RETIRED 0 empty never used, see retired, has been removed
retired files copied elsewhere, ready to abandon

Removed RETIRED today 2013-08-11

Three top level links remain to RETIRING :

  • d210 ( databasedumps, database_dumps_old, DriftWeb (small), rsync
    • keep DriftWeb link, and touch up this copy to /minos/data/afs/d210/DriftWeb
  • log_data 5 GB
    • attend to farm and bluwatch
  • release_data 3.4 GB - DONE
    • archived to /minos/data/afs/release_data 08/14
    • removed AFS data/minos/RETIRING/release_data Wed Aug 14 18:10:16 CDT 2013

2013-07-30 fsus-minos02 final cleanup

SHIFTED WEBDOCS 2013-07-30

cd /afs/fnal.gov/files/expwww/numi/html/offline_software
tokens

WDW=/afs/fnal.gov/files/code/e875/general/WebDocs

mv srt_public_context srt_public_context_DISABLED
mkdir srt_public_context
ln -s ${WDW} srt_public_context/WebDocs
date
Tue Jul 30 09:54:52 CDT 2013

Webdocs links are still working, at
http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/WebDocs_nav.html

2013-07-29 fsus-minos02 final cleanup

There are too many internal references in WebDocs to http://www-numi
to be corrected in the near future.

The total WebDocs area is about 600 MB, 533 MB of which is doxygen.

We can put these files in
/afs/.fnal.gov/files/code/e875/general/WebDocs,
rsyncing nightly from
/grid/fermiapp/minos/minossoft/releases/development/WebDocs

Doing this under the kreymer account, in order to have an AFS token.

Commissioned in minos27.kreymer
30 01 * * * /usr/krb5/bin/kcron ${HOME}/minos/scripts/webdocsync

Log is
/minos/data/users/kreymer/log/webdocsync/webdocsync.log

SHIFTED WEBDOCS 2013-07-30

cd /afs/fnal.gov/files/expwww/numi/html/offline_software
tokens

WDW=/afs/fnal.gov/files/code/e875/general/WebDocs

mv srt_public_context srt_public_context_DISABLED
mkdir srt_public_context
ln -s ${WDW} srt_public_context/WebDocs

date

2013-07-01 fsus-minos02 final cleanup

The last bit of AFS in use offline is WebDocs, updated by the AFS development build.
Robert Hatcher has just rsync'd this to /nusoft/app/web/htdoc/minos/WebDocs.
For example, http://nusoft.fnal.gov/minos/WebDocs/HEAD/doxygen/loon/html/index.html
We will put this in the rebuild cron job, redirect public links,
and stop the nightly AFS development build later this week.

Then we can proceed with the final cleanup.

  • Update all WebDocs links to nusoft
  • copy files from general to general2
  • remount all fsus-minos02 directories under RETIRED
  • mount general2 as general

2013-07-01 fsus-minos02 final cleanup

MOUNT VOL USED GB BLUE
general code.e875.general 5 / 8
general/minossoft nb.minos.d120 40 / 50 /grid/fermiapp/minos/minossoft/
general/ups nb.minos.d119 47 / 50 /grid/fermiapp/minos/products/
general/products c.e875.d1 3 / 8 /grid/fermiapp/minos/products/
releases nb.minos.d133 44 / 50 /grid/fermiapp/minos/minossoft/releases
releases1 nb.releases1 7 / 8 /grid/fermiapp/minos/minossoft/releases
releases2 releases2 6 / 8 /grid/fermiapp/minos/minossoft/releases

2013-06-28 fsus-minos02 final cleanup

Recorded code mount points ( shallow scan 4 deep confirmed by later full scan )


cd /afs/fnal.gov/files/code/e875

CMPS=`find . -maxdepth 4 -type d -exec fs lsmount {} \; 2>&1 | grep 'is a mount'`

printf "${CMPS}\n" > /minos/data/users/kreymer/log/afs/CMPS

Recorded hosts serving these mount points

CCDIRS=`cat /minos/data/users/kreymer/log/afs/CMPS | cut -f 2 -d "'"`
printf "${CDIRS}\n" > /minos/data/users/kreymer/log/afs/CDIRS

CSRVS=`for DIR in ${CDIRS} ; do fs whereis ${DIR} ; done`
printf "${CSRVS}\n" > /minos/data/users/kreymer/log/afs/CSRVS

File ./general is on host fsus-minos02.fnal.gov 
File ./general/minossoft is on host fsus-minos02.fnal.gov 
File ./general/ups is on host fsus-minos02.fnal.gov 
File ./general/products is on host fsus-minos02.fnal.gov 
File ./releases is on host fsus-minos02.fnal.gov 
File ./releases1 is on host fsus-minos02.fnal.gov 
File ./releases2 is on host fsus-minos02.fnal.gov 

File ./sim is on host fsus07.fnal.gov 
File ./rhbob is on host fsus07.fnal.gov 
File ./general2 is on host fsus02.fnal.gov 

2012 07 31 Volumes needing migration to other AFS servers :

Volume Mounted Usage Size
code.e875.general /afs/fnal.gov/files/code/e875/general software 5 GB
nb.minos.d119 /afs/fnal.gov/files/code/e875/general/ups old products 47 GB
nb.minos.d120 /afs/fnal.gov/files/code/e875/general/minossoft old releases 40 GB
nb.minos.d133 /afs/fnal.gov/files/code/e875/releases old releases 47 GB
nb.releases1 /afs/fnal.gov/files/code/e875/releases1 old releases 7 GB
nb.releases2 /afs/fnal.gov/files/code/e875/releases2 old releases 7 GB
minos.log_data /afs/fnal.gov/files/data/minos/log_data farm, farm_temp 1 GB
nb.minos.d199 /afs/fnal.gov/files/data/minos/calibration DriftWeb 5 MB
nb.minos.d210 /afs/fnal.gov/files/code/data/d210/rsync DBMAuto files 16 GB

Short term volume needs replicate the above, pending cleanup later in CY 2012

VOLUME GB MOUNT
minos.gen 8
minos.ups 50
minos.soft 50
minos.rel 50
minos.rel1 8
minos.rel2 8
minos.dat 50

After retirement of DBMAuto and the old software in ups/minossoft/releases* we should need no more than

VOLUME GB MOUNT
minos.general 8
minos.data 8

2012 07 06 aclscan of all data areas :

MAFS=/afs/.fnal.gov/files/data/minos

DIRS=`ls ${MAFS} | grep -v retired | sort`

{ date
for DIR in ${DIRS} ; do 
    printf "\nDIRECTORY ${DIR}\n" 
    SUBS=`find "${DIR}" -type d`
    printf "${SUBS}\n" | while read SUB; do
        PAR=`dirname "${SUB}"`
        fs listacl "${SUB}" | tail -n +2 > /tmp/sacl
        fs listacl "${PAR}" | tail -n +2 > /tmp/pacl
        if  ! diff -q /tmp/sacl /tmp/pacl >/dev/null ; then
            echo
            printf "${PAR}\n${SUB}\n" 
            sdiff -s /tmp/pacl /tmp/sacl
            #cat /tmp/pacl
            #cat /tmp/sacl
        fi
    done
done ; date; } 2>&1 | tee /minos/data/users/kreymer/log/afs/aclscan2.log

2012 07 05

  • Removed old parrot links from /afs/fnal.gov/files/expwww/numi/html/computing
    • rm d04libs d119 d120 d141 d199

2012 06 28

  • Preparing for directory ACL scan, surveyed paths with a large number of directories :
    • /minos/data/users/kreymer/log/afs/dircount.log

      Large areas and issues are :

/afs/fnal.gov/files/code/e875/general
d119 106938 - linked by /afs/fnal.gov/files/code/e875/general/ups
d120 - linked by /afs/fnal.gov/files/code/e875/general/minossoft
d141 149947 - another Parrot area, UPS products only, 2008

d162 13716 - MINOS_ROOT 12792

d186 find: d186/production/log: Permission denied
find: d186/production/BField: Permission denied
find: d186/production/CandFitTrackCam: Permission denied

d243 24504 - Griffin_Production_v00_Trimmed 24479

d245 24481 - Griffin_Production_v00_Trimmed 24467

d273 find: d273/kasahara: Permission denied

d86 26172 Minos code distribution work files, rhatcher

The largest code areas, d119, d120,d141 have been retired

2012 06 26

Full list of mount point under the data/minos tree :

MPS=`find . -type d -exec fs lsmount {} \; 2>&1 | grep 'is a mount'`

MINOS27 > printf "${MPS}\n" | wc -l
231

printf "${MPS}\n" > /minos/data/users/kreymer/log/afs/MPS

  • mount points not at the top :
    • './logbook/d1' is a mount point for volume '#minos.logbook.1'
    • './logbook/d2' is a mount point for volume '#minos.logbook.2'
    • './crl_data/CRLdata' is a mount point for volume '#d.minos.d2' - retired to /minos/data/maint/CRL
    • './crl_data/WWWdirectory' is a mount point for volume '#d.minos.d3' - retired to /minos/data/maint/CRL
    • './beam_data2/d1' is a mount point for volume '#nb.d.beamdata.d3' - retired to /minos/data/ 6/26
    • './beam_data2/d2' is a mount point for volume '#nb.d.beamdata.d4' - retired 6/26

{{last_modified}} MIGRATION NOTES -

We are copying files to Bluearc from volumes presently served by fsus-minos02, so this system can be shut down.

The copying will continue through the end of August

  • Detailed plan
    • Remount all the /afs/fnal.gov/files/data/minos/* directories under /afs/fnal.gov/files/data/minos/RETIRING/*
    • Install symlinks to the RETIRING/* path at the original paths.
    • Move one directory at a time to /minos/data/afs/*, moving to symlink to /minos/data/afs/* when the copy is verified.
    • Handle the special requests ( log_data, calibration/DriftWeb ) later, week of Aug 20.
  • Progress notes :
    • Aug 14 14:30 - Copied beam_docs db_cache dbm farm_logs farm_mclogs logbook1 logbook2 offline_monitor validation
    • Aug 15 12:40 - started moving d2?? directories. Rate seems to be 2 hours per 50 GB volume.
    • Aug 17 12:31 - started moving d??. Most are 8 GB, d77-d90 are 50 GB.
    • Aug 24 17:45 - most of calibration is moved
      • FdLinearityWorkspace took 6 hours to copy, 4 to diff
      • DriftWeb will be quick, but will do this Monday.
      • I have modified the copy procedure to avoid diff'ing symlinks. This may speed things up.
    • Aug 25 14:34 - Started moves of d1.. , starting with d106
    • Aug 27 14:00 - Moved calibration/DriftWeb to a new AFS location ( RETIRING/d210/DriftWeb )
    • Aug 28 11:37 - did a 'volrelease' to push .fnal.gov changes onto fnal.gov.
      • DriftWeb updates at 05:30 went to RETIRED/calibration/DriftWeb, probably due to lack of volrelese.
      • Check that these go to RETIRED/d210/DriftWeb tomorrow

A few more volumes mounted at and near the top of /afs/fnal.gov/files/code/e875


MINOS27 > fs whereis /afs/fnal.gov/files/code/e875/* | grep fsus-minos
File /afs/fnal.gov/files/code/e875/general is on host fsus-minos02.fnal.gov 
File /afs/fnal.gov/files/code/e875/releases is on host fsus-minos02.fnal.gov 
File /afs/fnal.gov/files/code/e875/releases1 is on host fsus-minos02.fnal.gov 
File /afs/fnal.gov/files/code/e875/releases2 is on host fsus-minos02.fnal.gov 

MINOS27 > fs whereis /afs/fnal.gov/files/code/e875/general/products
File /afs/fnal.gov/files/code/e875/general/products is on host fsus-minos02.fnal.gov 

MINOS27 > fs listquota /afs/fnal.gov/files/code/e875/*
code.e875.general            8000000    4979810   62%         23%  
nb.minos.d133               50000000   43171239   86%         39%  
nb.releases1                 8000000    7189834   90%         18%  
releases2                    8000000    6413858   80%         18%  

MINOS27 > fs listquota /afs/fnal.gov/files/code/e875/general/products
c.e875.d1                    8000000    3214256   40%         23%