Task #11056
Reshooting cmsstor411
Description
cmsstor411 had early warning of system disk error.
Warranty service called was placed on Monday 11/30/2015.
KOI service personnel came in and replaced the disk on Tuesday 12/01/2015 without notifying us.
As now, Wednesday, 12/02/2015, cmsstor411 is still down.
Tried to reshoot it by placing it under repair mode.
That did not trigger the install.
Will try something else to reshoot it.
History
#1 Updated by Chih-Hao Huang about 5 years ago
[1] create a branch huangch_cmsstor411_reshoot
[2] assign w-cmsstor411-disk-disk1 and w-cmsstor411-disk-disk2 to readonlyPools group
1:44pm phoebehannah:~/GIT/puppet> make_branch huangch_cmsstor411_reshoot
Changing to the top level directory [/Users/huangch/GIT/puppet]
Creating huangch_cmsstor411_reshoot branch.
Switching to the itb branch. This way we ensure the new branch is based on itb
Already on 'itb'
Your branch is up-to-date with 'origin/itb'.
Switched to a new branch 'huangch_cmsstor411_reshoot'
Total 0 (delta 0), reused 0 (delta 0)
remote: fatal: bad object 0000000000000000000000000000000000000000
remote:
remote: diff-tree:
remote: Recieved from stdin:
remote: oldrev: 0000000000000000000000000000000000000000
remote: newrev: 727ac88665b48b3e41f62ba632e893a4669dc564
remote: refname: refs/heads/huangch_cmsstor411_reshoot
remote: Derived Configuration:
remote: REPO: puppet@cms-git:/var/lib/puppet/puppet.git
remote: BRANCH: huangch_cmsstor411_reshoot
remote: BRANCH_DIR: /srv/puppet/environments
remote: PUPPET_SERVERS: puppet@cmssrv166.fnal.gov puppet@cmspuppet2.fnal.gov puppet@cmspuppet1.fnal.gov
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmssrv166.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: couldn't update huangch_cmsstor411_reshoot, will try to create new environment
remote: Initialized empty Git repository in /srv/puppet/environments/huangch_cmsstor411_reshoot/.git/
remote: Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
remote: Switched to a new branch 'huangch_cmsstor411_reshoot'
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet2.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: couldn't update huangch_cmsstor411_reshoot, will try to create new environment
remote: Initialized empty Git repository in /srv/puppet/environments/huangch_cmsstor411_reshoot/.git/
remote: Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
remote: Switched to a new branch 'huangch_cmsstor411_reshoot'
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet1.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: couldn't update huangch_cmsstor411_reshoot, will try to create new environment
remote: Initialized empty Git repository in /srv/puppet/environments/huangch_cmsstor411_reshoot/.git/
remote: Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
remote: Switched to a new branch 'huangch_cmsstor411_reshoot'
To puppet@dcso-git.fnal.gov:/var/lib/puppet/puppet.git
* [new branch] huangch_cmsstor411_reshoot > huangch_cmsstor411_reshoot a/modules/dcache/files/etc/dcache/poolmanager-disk.conf
Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
1:47pm phoebehannah:~/GIT/puppet> cd modules/dcache/files/etc/dcache
1:48pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> vi poolmanager-disk.conf
1:49pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git diff
diff --git a/modules/dcache/files/etc/dcache/poolmanager-disk.conf b/modules/dcache/files/etc/dcache/poolmanager-disk.con
index 08dfb58..eb7d910 100644
--
++ b/modules/dcache/files/etc/dcache/poolmanager-disk.conf@ -1087,8 +1087,8 @ psu addto pgroup flushPools w-cmsstor409-disk-disk1
psu addto pgroup flushPools w-cmsstor409-disk-disk2
psu addto pgroup flushPools w-cmsstor410-disk-disk1
psu addto pgroup flushPools w-cmsstor410-disk-disk2psu addto pgroup flushPools w-cmsstor411-disk-disk1), 2 deletions()
-psu addto pgroup flushPools w-cmsstor411-disk-disk2
+psu addto pgroup readonlyPools w-cmsstor411-disk-disk1
+psu addto pgroup readonlyPools w-cmsstor411-disk-disk2
psu addto pgroup flushPools w-cmsstor412-disk-disk1
psu addto pgroup flushPools w-cmsstor412-disk-disk2
psu addto pgroup flushPools w-cmsstor413-disk-disk1
1:56pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git add poolmanager-disk.conf
1:57pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git commit
[huangch_cmsstor411_reshoot b121e97] assign w-cmsstor411-disk-disk1 and w-cmsstor411-disk-disk2 to readonlyPools group for reshooting cmsstor411
1 file changed, 2 insertions(
1:58pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache>
2:02pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git push
Counting objects: 43, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 821 bytes | 0 bytes/s, done.
Total 8 (delta 5), reused 0 (delta 0)
remote:
remote: diff-tree:
remote: :100644 100644 08dfb5869d370ff1d5ee823dd43449a52d2f572d eb7d910b1e4b3e7e6352017ae7d3aafc802205ea M modules/dcache/files/etc/dcache/poolmanager-disk.conf
remote: Recieved from stdin:
remote: oldrev: 727ac88665b48b3e41f62ba632e893a4669dc564
remote: newrev: b121e977e6e2aaf60b4f5d74d508c11af227d188
remote: refname: refs/heads/huangch_cmsstor411_reshoot
remote: Derived Configuration:
remote: REPO: puppet@cms-git:/var/lib/puppet/puppet.git
remote: BRANCH: huangch_cmsstor411_reshoot
remote: BRANCH_DIR: /srv/puppet/environments
remote: PUPPET_SERVERS: puppet@cmssrv166.fnal.gov puppet@cmspuppet2.fnal.gov puppet@cmspuppet1.fnal.gov
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmssrv166.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: From cms-git:/var/lib/puppet/puppet
remote: * branch huangch_cmsstor411_reshoot > FETCH_HEAD
remote: Updating 727ac88..b121e97
remote: Fast-forward
remote: .../dcache/files/etc/dcache/poolmanager-disk.conf | 4 +-
remote: 1 files changed, 2 insertions(), 2 deletions()
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet2.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: From cms-git:/var/lib/puppet/puppet
remote: * branch huangch_cmsstor411_reshoot -> FETCH_HEAD
remote: Updating 727ac88..b121e97
remote: Fast-forward
remote: .../dcache/files/etc/dcache/poolmanager-disk.conf | 4 +-
remote: 1 files changed, 2 insertions(), 2 deletions()
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet1.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: From cms-git:/var/lib/puppet/puppet
remote: * branch huangch_cmsstor411_reshoot -> FETCH_HEAD
remote: Updating 727ac88..b121e97
remote: Fast-forward
remote: .../dcache/files/etc/dcache/poolmanager-disk.conf | 4 +-
remote: 1 files changed, 2 insertions(), 2 deletions(-)
To puppet@dcso-git.fnal.gov:/var/lib/puppet/puppet.git
727ac88..b121e97 huangch_cmsstor411_reshoot -> huangch_cmsstor411_reshoot
2:02pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache>
[3] put cmsdcacheadmindisk on this branch
#2 Updated by Chih-Hao Huang about 5 years ago
- % Done changed from 0 to 30
#3 Updated by Chih-Hao Huang about 5 years ago
- % Done changed from 30 to 60
[root@cmsadmin1 ~]# cms-shoot cmsstor411
removing host from rocks on cmsrocks51, if necessary
cmsstor24.fnal.gov: no host cmsstor411 to remove
Connection to cmsrocks51 closed.
removing host from rocks on cmsrocks52, if necessary
cmssrv26.fnal.gov: no host cmsstor411 to remove
Connection to cmsrocks52 closed.
stopping puppet on cmsstor411, if applicable
ssh: connect to host cmsstor411 port 22: Connection refused
telling host to netboot on next boot
cmsstor411: netboot -> True
set 1 hosts to boot
1 system(s) updated
telling cmspuppetca to remove host's cert, if present
cleaning cert for cmsstor411.fnal.gov
Notice: Revoked certificate with serial 3238
Notice: Removing file Puppet::SSL::Certificate cmsstor411.fnal.gov at '/var/lib/puppet/ssl/ca/signed/cmsstor411.fnal.gov.pem'
Notice: Removing file Puppet::SSL::Certificate cmsstor411.fnal.gov at '/var/lib/puppet/ssl/certs/cmsstor411.fnal.gov.pem'
telling cmspuppetca to update autosign information
when you're ready to start, run:
cmspower-powerit --action cycle --comment 'reinstalling' cmsstor411
don't forget to disable zabbix monitoring if applicable
[root@cmsadmin1 ~]# cmspower-powerit --action cycle --comment 'reinstalling' cmsstor411
/usr/bin/ssh -l root cmsconsole cmspower-powerit --action cycle --comment \'root: reinstalling\' cmsstor411
Outlet state: OFF
Outlet state: ON
=== cmsstor411 ===
connecting to APC apccms1015-1, outlet 4
connecting to APC apccms1015-1, outlet 4
[root@cmsadmin1 ~]#
#4 Updated by Chih-Hao Huang about 5 years ago
- Status changed from Assigned to Resolved
- % Done changed from 60 to 100
It is up and everything looks fine.