Project

General

Profile

Support #15794

Milestone #15057: Minos SLF5 retirement

Support #15792: Individual Minos SLF5 node shutdowns

minos-sam04 retirement

Added by Arthur Kreymer over 3 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
High
Start date:
03/08/2017
Due date:
03/31/2017
% Done:

100%

Estimated time:
10.00 h
Duration: 24

Description

Retire minos-sam04

History

#1 Updated by Arthur Kreymer over 3 years ago

Summary table and overall notes

Task Completed Comment
Ganglia 03/16 Idle
procsum 03/16 Idle since before 03/10
home 03/16 copied parts to /minos/data/mindata/archive/minos-sam04
Ganglia monthly averages
Idle 99.8%
Load 60m
Net in  <1k
Net out 3k

#2 Updated by Arthur Kreymer over 3 years ago

  • % Done changed from 10 to 20

SAM cleanup

procsum shows sam account activity

 Count of /var/log/prochistory files 7201
20170303_19:23:01_procs.gz
20146 sam       17   0  107m 2012 1228 S  0.0  0.0   0:00.00 crond
20151 sam       21   0  8728 1056  876 S  0.0  0.0   0:00.00 /bin/sh -c . ${HOME}/setups.sh &&  ${HOME}/maint/nameservice/name_service_cleanup prd && ${HOME}/maint/nameservice/name_service_cleanup int && ${HOME}/maint/nameservice/name_service_cleanup dev
20459 sam       25   0  8728 1068  880 S  0.0  0.0   0:00.00 /bin/sh /home/sam/maint/nameservice/name_service_cleanup int

I verified with mengel and illingwo that minos-sam04 is no longer used by SAM.

I removed the sam crontab entry :

23 1,7,13,19 * * * . ${HOME}/setups.sh &&  ${HOME}/maint/nameservice/name_service_cleanup prd && ${HOME}/maint/nameservice/name_service_cleanup int && ${HOME}/maint/nameservice/name_service_cleanup dev

#33 3 1 * * ${HOME}/private/archive/archsamlogs 

#@reboot  sleep 60 ; until [ -d $HOME ] ; do sleep 1; done;  ${HOME}/samstart

37 */3 * * * ${HOME}/bin/make_proxy.sh

The proxies being made in /var/tmp are not valid past last October.
The name servers are no longer running.

crontab -l > crontab.20170309
ln -sf crontab.20170309 crontab.dat # was crontab.20130516
date ; crontab -r

Will check system activity again tomorrow.

#3 Updated by Arthur Kreymer over 3 years ago

LOCAL FILES

Local files are all in home

/opt/mindata and minospro are empty

df -h | grep dev
/dev/sda1              20G  5.0G   14G  27% /
/dev/sda3             2.0G   44M  1.8G   3% /tmp
/dev/sda5             201G  1.7G  189G   1% /var
tmpfs                 7.9G  4.0K  7.9G   1% /dev/shm
/dev/sdc1             230G  201G   18G  93% /home

drwxr-xr-x  5 mindata e875  4096 May 20  2011 mindata/
drwxr-xr-x  7 minfarm e875  4096 Jul  4  2011 minfarm/
drwxr-xr-x  8 minsoft   27  4096 Mar  8  2013 minsoft/
drwxr-xr-x  3 root    root  4096 Nov 17  2008 room1/
drwxr-xr-x 15 sam     dbi   4096 Mar  9 12:07 sam/
drwxr-xr-x  5 samread dbi   4096 Nov 10  2014 samread/

mindata - nothing of interest, removed 03/16
rm -r /home/mindata

minfarm - nothing of interest, vintage 2009, removed 03/16
rm -r /home/minfarm

samread - nothing of interest, abandon 2014 dbserver logs 2014 04 to 11

minsoft - products tests, copied to /minos/data/

 du -sm /home/minsoft/
183    /home/minsoft/

MS=/grid/fermiapp/minos/scripts/modescan
${MS} /home/minsoft
${MS} /home/minsoft fix

LOG in /var/tmp/modescan-minsoft.log
File counts : 
    gr 5
    gx 1
    or 59
    ox 7
    uw 0

mkdir /minos/data/mindata/archive/minos-sam04
cp -ax  /home/minsoft  /minos/data/mindata/archive/minos-sam04/minsoft
diff -r /home/minsoft  /minos/data/mindata/archive/minos-sam04/minsoft
diff: /home/minsoft/AMS: No such file or directory
...
du -sm  /home/minsoft  /minos/data/mindata/archive/minos-sam04/minsoft
183    /home/minsoft
350    /minos/data/mindata/archive/minos-sam04/minsoft

find /home/minsoft -type f | wc -l
6646
find /minos/data/mindata/archive/minos-sam04/minsoft -type f | wc -l
6646

sam - large areas are
183921 private
8822 products

Removed private log areas

du -sm station__minos*
1386    station__minos-sam04__coupp_prd__coupp
38    station__minos-sam04__d0_dcache__d0_dcache
20    station__minos-sam04__darkside_prd__darkside
109    station__minos-sam04__gm2_prd__gm2
15035    station__minos-sam04__lariat_prd__lariat
19    station__minos-sam04__lbne_prd__lbne
1    station__minos-sam04__lbne_prd__lbne-int
2270    station__minos-sam04__minerva_prd__minerva
22    station__minos-sam04__minerva_prd__minerva-dcache
367    station__minos-sam04__minos_prd__minos
1    station__minos-sam04__mu2e_dev__mu2e
113189    station__minos-sam04__mu2e_prd__mu2e
36695    station__minos-sam04__nova_prd__nova
2351    station__minos-sam04__nova_prd__nova-int
1    station__minos-sam04__station_dev__minos
1    station__minos-sam04__station_prd__minos
1    station__minos-sam04__station_prd__minos-int
12398    station__minos-sam04__uboone_prd__uboone
1    station__minos-sam04__uboone_prd__uboone-int

rm -r station__minos*

Moving the rest

MS=/grid/fermiapp/minos/scripts/modescan
${MS} /home/sam
${MS} /home/sam fix

    modescan 20140512 testing file permissions
Thu Mar 16 10:33:21 CDT 2017
File counts : 
    gr 31
    gx 3
    or 50
    ox 9
    uw 503

Thu Mar 16 10:33:55 CDT 2017
 Setting g+r permissions 
Thu Mar 16 10:33:55 CDT 2017
 Setting g+x permissions 
Thu Mar 16 10:33:55 CDT 2017
 Setting o+r permissions 
Thu Mar 16 10:33:56 CDT 2017
 Setting o+x permissions 
Thu Mar 16 10:33:56 CDT 2017
 Setting u+w permissions 
Thu Mar 16 10:33:57 CDT 2017

cp -ax  /home/sam  /minos/data/mindata/archive/minos-sam04/sam

diff -r /home/sam  /minos/data/mindata/archive/minos-sam04/sam
diff: /home/sam/LOG: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/LOG: No such file or directory
diff: /home/sam/products/oracle_tnsnames/v77/NULL/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/oracle_tnsnames/v77/NULL/ups/..tar: No such file or directory
diff: /home/sam/products/sam/v8_6_3/Linux+2/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/sam/v8_6_3/Linux+2/ups/..tar: No such file or directory
diff: /home/sam/products/sam/v8_8_4/Linux+2/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/sam/v8_8_4/Linux+2/ups/..tar: No such file or directory
diff: /home/sam/products/sam_ns_ior/v7_2_2/NULL/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/sam_ns_ior/v7_2_2/NULL/ups/..tar: No such file or directory
diff: /home/sam/products/sam_ns_ior/v7_2_3/NULL/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/sam_ns_ior/v7_2_3/NULL/ups/..tar: No such file or directory
diff: /home/sam/products/upd/v4_7_4a/NULL/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/upd/v4_7_4a/NULL/ups/..tar: No such file or directory
diff: /home/sam/products/upd_libs/v5_8_5/Linux+2/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/upd_libs/v5_8_5/Linux+2/ups/..tar: No such file or directory
diff: /home/sam/products/ups/v4_9_2/Linux+2/ups/..tar: No such file or directory
diff: /minos/data/mindata/archive/minos-sam04/sam/products/ups/v4_9_2/Linux+2/ups/..tar: No such file or directory
diff: /home/sam/products/upsdb/sam/Symlinks/current/v8_8_4: Too many levels of symbolic links
diff: /minos/data/mindata/archive/minos-sam04/sam/products/upsdb/sam/Symlinks/current/v8_8_4: Too many levels of symbolic links
diff: /home/sam/products/upsdb/sam_config/Symlinks/current/v7_1_7: Too many levels of symbolic links
diff: /minos/data/mindata/archive/minos-sam04/sam/products/upsdb/sam_config/Symlinks/current/v7_1_7: Too many levels of symbolic links
diff: /home/sam/products/upsdb/sam_ns_ior/Symlinks/current/v7_2_1: Too many levels of symbolic links
diff: /minos/data/mindata/archive/minos-sam04/sam/products/upsdb/sam_ns_ior/Symlinks/current/v7_2_1: Too many levels of symbolic links
diff: /home/sam/products/upsdb/sam_ns_ior/Symlinks/current/v7_2_3: Too many levels of symbolic links
diff: /minos/data/mindata/archive/minos-sam04/sam/products/upsdb/sam_ns_ior/Symlinks/current/v7_2_3: Too many levels of symbolic links

du -sm  /home/sam  /minos/data/mindata/archive/minos-sam04/sam
9048    /home/sam
12934    /minos/data/mindata/archive/minos-sam04/sam

find  /home/sam -type f | wc -l
157005
find  /minos/data/mindata/archive/minos-sam04/sam -type f | wc -l
157005

#4 Updated by Arthur Kreymer over 3 years ago

  • % Done changed from 20 to 80

RITM0539760 03/16 minos-sam04 shutdown

At your convenience, please shut down the minos-sam04 system.
This SLF5 system is being retired.

See preparation details in https://cdcvs.fnal.gov/redmine/issues/15794

Please hold the hardware for a month before disposal,
in case we overlooked something needing archival.

2017-03-16 09:58:08 CDT - Christophe Bonnaud
Minos25 has been shutdown.

#5 Updated by Arthur Kreymer over 3 years ago

  • Status changed from Work in progress to Resolved
  • % Done changed from 80 to 100

#6 Updated by Arthur Kreymer about 3 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF