Project

General

Profile

Support #15796

Milestone #15057: Minos SLF5 retirement

Support #15792: Individual Minos SLF5 node shutdowns

minos27 retirement

Added by Arthur Kreymer over 3 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
High
Start date:
03/08/2017
Due date:
03/31/2017
% Done:

100%

Estimated time:
20.00 h
Duration: 24

Description

Retire minos27 by 2017 April 1

History

#1 Updated by Arthur Kreymer over 3 years ago

Summary

Task Completed Comment
Ganglia 03/01 Idle
procsum 03/19 Users off 03/, idle from 03/
scratch27 03/18 to /minos/data/mindata/archive/minos27/scratch27
home 03/18 nothing there
opt 03/18 nothing needed

Ganglia averages
Idle 99.7%
Load 1 49m
Net in 7k
Net out 9k

#2 Updated by Arthur Kreymer over 3 years ago

  • % Done changed from 10 to 50

Date: Sat, 18 Mar 2017 13:26:52 +0000
From: Arthur Kreymer <>
To:
Subject: minos-dcs02-nd bin/ToBluearc.sh disabled

I have taken the liberty of disabling minos-dcs-02 bin/ToBluearc.sh

This script was rsync'ing TOF data through minos27.

We are shutting down minos27 soon.
The latest TOF file is dated Aug 10 2016.

It would be good to remove the crontab entry
0 20 * * * /home/minos/bin/ToBluearc.sh

#3 Updated by Arthur Kreymer over 3 years ago

Local files are archived

   df -hP | grep ^/dev

/dev/sda1              20G  5.0G   14G  27% /
/dev/sda6             191G   16G  165G   9% /var
/dev/sda5             2.0G   45M  1.8G   3% /tmp
/dev/sda2             9.7G  151M  9.1G   2% /home
/dev/mapper/minos-local  932G   15G  917G   2% /local/scratch27
HOME is empty OPT
r------- 1 minospro e875 8353 Mar 18 05:58 minospro.Production.proxy
rw------ 1 mindata e875 72 Jan 24 15:53 kreymer-cron-minos27.keytab

The kreymer keytab is also on minos-data.

Cancelled the minospro.Producion proxy push with
RITM0540413 03/18 minospro managed proxy removal from minos27 and minos51

SCRATCH27
drwxrwxrwt  9 root     root  138 Apr  2  2015 ./
drwxr-xr-x  4 root     root 4096 Jan 12  2012 ../
drwxr-xr-x  2 corwin   e875    6 Jul  9  2011 corwin/
drwxr-xr-x  3 jdejong  e875   34 Oct 30  2011 jdejong/
drwxr-xr-x 10 kreymer  e875 4096 Apr 22  2015 kreymer/
drwxr-xr-x  8 mindata  e875  123 Jul 11  2014 mindata/
drwxr-xr-x  3 minosdb  e875   17 Dec 11  2012 minosdb/
d--x--x-wx  2 mindata  e875    6 Aug 30  2012 MINOSgains/
drwxr-xr-x  4 rhatcher e875   45 Nov 10  2011 rhatcher/
-rw-rw-r--  1 terlyga  e875  387 Sep 20  2012 temp.txt
-rw-rw-r--  1 terlyga  e875  687 Sep 20  2012 temp.txt~

du -sm *
0    corwin
616    jdejong
6245    kreymer
5880    mindata
1    minosdb
1571    rhatcher
1    temp.txt
1    temp.txt~

Removed empty MINOSgains directory

Copied files

SIN=/local/scratch27
SOU=/minos/data/mindata/archive/minos27
SOUT=/minos/data/mindata/archive/minos27/scratch27

mkdir -p ${SOU}
date ; cp -ax ${SIN} ${SOU}/scratch27 2>&1 | tee /tmp/scratch27 ; date
Sat Mar 18 09:09:00 CDT 2017
cp: cannot open `/local/scratch27/rhatcher/run_reco.1802/Production/Dogwood/core.2444' for reading: Permission denied
cp: cannot open `/local/scratch27/rhatcher/run_reco.alt/Production/Dogwood/core.2444' for reading: Permission denied
Sat Mar 18 09:54:40 CDT 2017

du -sm ${SIN} ${SOUT}
13982    /local/scratch27
45752    /minos/data/mindata/archive/minos27/scratch27

find ${SIN} -type f | wc -l
808564
find ${SOUT} -type f | wc -l
808562

diff -r ${SIN} ${SOUT}
Only in /local/scratch27/rhatcher/run_reco.1802/Production/Dogwood: core.2444
Only in /local/scratch27/rhatcher/run_reco.alt/Production/Dogwood: core.2444

#4 Updated by Arthur Kreymer over 3 years ago

collector file copies

RITM0538920 03/15 ifbeam collector file copies to minos27

We are about to shut down the SLF5 minos27 host.

We see what appear to be rsync operations from dbweb5 to minos27
writing to /minos/app/mindata/export/collectors/long.

Please redirect these to host minos-data.
________________________________

2017-03-18 13:58:25 CDT - Vladimir Podstavkov (Additional comments)
reply from:
 
Done!
______________________________

./procsum today 
grep collectors  /minos/data/web/computing/dh/procsum/minosdatagpvm01/minosdatagpvm01-20170318

less  /minos/data/web/computing/dh/procsum/minos27/minos27-20170318
...
20170318_13:35:01_procs.gz
 1171 mindata   17   0 66096  892  724 S  0.0  0.0   0:00.00 rsync --server -te.Ls . /minos/app/mindata/export/collectors/long
 1174 mindata   17   0 66096  892  724 S  0.0  0.0   0:00.00 rsync --server -te.Ls . /minos/app/mindata/export/collectors/long
 1191 mindata   18   0 66356  692  220 D  0.0  0.0   0:00.00 rsync --server -te.Ls . /minos/app/mindata/export/collectors/long
 1192 mindata   18   0 66096  376  196 S  0.0  0.0   0:00.00 rsync --server -te.Ls . /minos/app/mindata/export/collectors/long

ganglia for minosdatagpvm01 is absent since March 16 09:30


Thanks !

New files are showing up in /minos/app/mindata/export/collectors/long
like
-rw-r--r-- 1 mindata e875  1020791 Mar 18 14:03 1489863678583.collect.NuMI_Monitoring.1489670951168.out.closed.gz

Ganglia shows network traffic on minos27 dropping around 13:55,
from 150k to near zero.

This RITM can be closed.

#5 Updated by Arthur Kreymer over 3 years ago

  • % Done changed from 50 to 80

#6 Updated by Arthur Kreymer over 3 years ago

Date: Mon, 20 Mar 2017 13:55:22 +0000
From: Arthur Kreymer <>
To:
Cc: , ,
Subject: bremple account using minos27

We are ready to shut down the old minos27 system.

We see continuing usage by the bremple account.
http://minos.fnal.gov/computing/dh/procsum/minos27/minos27-20170319

Bryce - Please use minos60 through minos63,
or minos-slf5 if you need to use a system using
the older SLF5 operating sytem.

Thanks !


sent this via wall.

Verified using jdejong account.

Broadcast message from kreymer (pts/0) (Mon Mar 20 08:57:58 2017):
_________________________________________________________________

Date: Mon, 20 Mar 2017 12:23:55 -0500
From: Alec T. Habig <>

Thanks Art!

He's using that machine because it was the one I used as an example :)

We're about to figure out how to push all the stuff off to the grid
anyway.

#7 Updated by Arthur Kreymer over 3 years ago

  • % Done changed from 80 to 90

RITM0541265 03/20 minos27 shutdown

At your convenience, please shut down the minos27 system.
This SLF5 system is being retired.

See preparation details in https://cdcvs.fnal.gov/redmine/issues/15796

Please hold the hardware for a month before disposal,
in case we overlooked something.
_________________________________________________________________

2017-03-21 09:27:09 CDT - Christophe Bonnaud (Additional comments)

Minos27 is now down.

#8 Updated by Arthur Kreymer over 3 years ago

  • Status changed from Work in progress to Resolved
  • % Done changed from 90 to 100

#9 Updated by Arthur Kreymer about 3 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF