Task #11056: Reshooting cmsstor411 - CMS dCache - Fermilab Redmine

Task #11056

Reshooting cmsstor411

Added by Chih-Hao Huang over 4 years ago. Updated over 4 years ago.

Status:

Resolved

Priority:

Normal

Assignee:

Chih-Hao Huang

Start date:

12/02/2015

Due date:

12/09/2015

% Done:

100%

Estimated time:

4.00 h

Spent time:

2.50 h

Duration: 8

Description

cmsstor411 had early warning of system disk error.
Warranty service called was placed on Monday 11/30/2015.
KOI service personnel came in and replaced the disk on Tuesday 12/01/2015 without notifying us.
As now, Wednesday, 12/02/2015, cmsstor411 is still down.

Tried to reshoot it by placing it under repair mode.
That did not trigger the install.

Will try something else to reshoot it.

History

#1 Updated by Chih-Hao Huang over 4 years ago

[1] create a branch huangch_cmsstor411_reshoot
[2] assign w-cmsstor411-disk-disk1 and w-cmsstor411-disk-disk2 to readonlyPools group

1:44pm phoebehannah:~/GIT/puppet> make_branch huangch_cmsstor411_reshoot
Changing to the top level directory [/Users/huangch/GIT/puppet]
Creating huangch_cmsstor411_reshoot branch.
Switching to the itb branch. This way we ensure the new branch is based on itb
Already on 'itb'
Your branch is up-to-date with 'origin/itb'.
Switched to a new branch 'huangch_cmsstor411_reshoot'
Total 0 (delta 0), reused 0 (delta 0)
remote: fatal: bad object 0000000000000000000000000000000000000000
remote:
remote: diff-tree:
remote: Recieved from stdin:
remote: oldrev: 0000000000000000000000000000000000000000
remote: newrev: 727ac88665b48b3e41f62ba632e893a4669dc564
remote: refname: refs/heads/huangch_cmsstor411_reshoot
remote: Derived Configuration:
remote: REPO: puppet@cms-git:/var/lib/puppet/puppet.git
remote: BRANCH: huangch_cmsstor411_reshoot
remote: BRANCH_DIR: /srv/puppet/environments
remote: PUPPET_SERVERS: puppet@cmssrv166.fnal.gov puppet@cmspuppet2.fnal.gov puppet@cmspuppet1.fnal.gov
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmssrv166.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: couldn't update huangch_cmsstor411_reshoot, will try to create new environment
remote: Initialized empty Git repository in /srv/puppet/environments/huangch_cmsstor411_reshoot/.git/
remote: Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
remote: Switched to a new branch 'huangch_cmsstor411_reshoot'
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet2.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: couldn't update huangch_cmsstor411_reshoot, will try to create new environment
remote: Initialized empty Git repository in /srv/puppet/environments/huangch_cmsstor411_reshoot/.git/
remote: Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
remote: Switched to a new branch 'huangch_cmsstor411_reshoot'
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet1.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: couldn't update huangch_cmsstor411_reshoot, will try to create new environment
remote: Initialized empty Git repository in /srv/puppet/environments/huangch_cmsstor411_reshoot/.git/
remote: Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
remote: Switched to a new branch 'huangch_cmsstor411_reshoot'
To puppet@dcso-git.fnal.gov:/var/lib/puppet/puppet.git * [new branch] huangch_cmsstor411_reshoot > huangch_cmsstor411_reshoot
Branch huangch_cmsstor411_reshoot set up to track remote branch huangch_cmsstor411_reshoot from origin.
1:47pm phoebehannah:~/GIT/puppet> cd modules/dcache/files/etc/dcache
1:48pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> vi poolmanager-disk.conf
1:49pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git diff
diff --git a/modules/dcache/files/etc/dcache/poolmanager-disk.conf b/modules/dcache/files/etc/dcache/poolmanager-disk.con
index 08dfb58..eb7d910 100644
-- a/modules/dcache/files/etc/dcache/poolmanager-disk.conf
++ b/modules/dcache/files/etc/dcache/poolmanager-disk.conf
@ -1087,8 +1087,8 @ psu addto pgroup flushPools w-cmsstor409-disk-disk1
psu addto pgroup flushPools w-cmsstor409-disk-disk2
psu addto pgroup flushPools w-cmsstor410-disk-disk1
psu addto pgroup flushPools w-cmsstor410-disk-disk2
psu addto pgroup flushPools w-cmsstor411-disk-disk1
-psu addto pgroup flushPools w-cmsstor411-disk-disk2
+psu addto pgroup readonlyPools w-cmsstor411-disk-disk1
+psu addto pgroup readonlyPools w-cmsstor411-disk-disk2
psu addto pgroup flushPools w-cmsstor412-disk-disk1
psu addto pgroup flushPools w-cmsstor412-disk-disk2
psu addto pgroup flushPools w-cmsstor413-disk-disk1
1:56pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git add poolmanager-disk.conf
1:57pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git commit
[huangch_cmsstor411_reshoot b121e97] assign w-cmsstor411-disk-disk1 and w-cmsstor411-disk-disk2 to readonlyPools group for reshooting cmsstor411
1 file changed, 2 insertions(), 2 deletions()
1:58pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache>
2:02pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache> git push
Counting objects: 43, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (8/8), done.
Writing objects: 100% (8/8), 821 bytes | 0 bytes/s, done.
Total 8 (delta 5), reused 0 (delta 0)
remote:
remote: diff-tree:
remote: :100644 100644 08dfb5869d370ff1d5ee823dd43449a52d2f572d eb7d910b1e4b3e7e6352017ae7d3aafc802205ea M modules/dcache/files/etc/dcache/poolmanager-disk.conf
remote: Recieved from stdin:
remote: oldrev: 727ac88665b48b3e41f62ba632e893a4669dc564
remote: newrev: b121e977e6e2aaf60b4f5d74d508c11af227d188
remote: refname: refs/heads/huangch_cmsstor411_reshoot
remote: Derived Configuration:
remote: REPO: puppet@cms-git:/var/lib/puppet/puppet.git
remote: BRANCH: huangch_cmsstor411_reshoot
remote: BRANCH_DIR: /srv/puppet/environments
remote: PUPPET_SERVERS: puppet@cmssrv166.fnal.gov puppet@cmspuppet2.fnal.gov puppet@cmspuppet1.fnal.gov
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmssrv166.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: From cms-git:/var/lib/puppet/puppet
remote: * branch huangch_cmsstor411_reshoot > FETCH_HEAD
remote: Updating 727ac88..b121e97
remote: Fast-forward
remote: .../dcache/files/etc/dcache/poolmanager-disk.conf | 4 +-
remote: 1 files changed, 2 insertions(), 2 deletions()
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet2.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: From cms-git:/var/lib/puppet/puppet
remote: * branch huangch_cmsstor411_reshoot -> FETCH_HEAD
remote: Updating 727ac88..b121e97
remote: Fast-forward
remote: .../dcache/files/etc/dcache/poolmanager-disk.conf | 4 +-
remote: 1 files changed, 2 insertions(), 2 deletions()
remote: Updating remote branch /srv/puppet/environments/huangch_cmsstor411_reshoot on puppet@cmspuppet1.fnal.gov
remote: attempting to pull branch huangch_cmsstor411_reshoot to /srv/puppet/environments/huangch_cmsstor411_reshoot
remote: From cms-git:/var/lib/puppet/puppet
remote: * branch huangch_cmsstor411_reshoot -> FETCH_HEAD
remote: Updating 727ac88..b121e97
remote: Fast-forward
remote: .../dcache/files/etc/dcache/poolmanager-disk.conf | 4 +-
remote: 1 files changed, 2 insertions(), 2 deletions(-)
To puppet@dcso-git.fnal.gov:/var/lib/puppet/puppet.git
727ac88..b121e97 huangch_cmsstor411_reshoot -> huangch_cmsstor411_reshoot
2:02pm phoebehannah:~/GIT/puppet/modules/dcache/files/etc/dcache>

[3] put cmsdcacheadmindisk on this branch

#2 Updated by Chih-Hao Huang over 4 years ago

% Done changed from 0 to 30

#3 Updated by Chih-Hao Huang over 4 years ago

% Done changed from 30 to 60

[root@cmsadmin1 ~]# cms-shoot cmsstor411
removing host from rocks on cmsrocks51, if necessary
cmsstor24.fnal.gov: no host cmsstor411 to remove
Connection to cmsrocks51 closed.
removing host from rocks on cmsrocks52, if necessary
cmssrv26.fnal.gov: no host cmsstor411 to remove
Connection to cmsrocks52 closed.
stopping puppet on cmsstor411, if applicable
ssh: connect to host cmsstor411 port 22: Connection refused
telling host to netboot on next boot
cmsstor411: netboot -> True
set 1 hosts to boot
1 system(s) updated
telling cmspuppetca to remove host's cert, if present
cleaning cert for cmsstor411.fnal.gov
Notice: Revoked certificate with serial 3238
Notice: Removing file Puppet::SSL::Certificate cmsstor411.fnal.gov at '/var/lib/puppet/ssl/ca/signed/cmsstor411.fnal.gov.pem'
Notice: Removing file Puppet::SSL::Certificate cmsstor411.fnal.gov at '/var/lib/puppet/ssl/certs/cmsstor411.fnal.gov.pem'
telling cmspuppetca to update autosign information
when you're ready to start, run:
cmspower-powerit --action cycle --comment 'reinstalling' cmsstor411
don't forget to disable zabbix monitoring if applicable
[root@cmsadmin1 ~]# cmspower-powerit --action cycle --comment 'reinstalling' cmsstor411
/usr/bin/ssh -l root cmsconsole cmspower-powerit --action cycle --comment \'root: reinstalling\' cmsstor411
Outlet state: OFF
Outlet state: ON === cmsstor411 ===
connecting to APC apccms1015-1, outlet 4
connecting to APC apccms1015-1, outlet 4
[root@cmsadmin1 ~]#

#4 Updated by Chih-Hao Huang over 4 years ago

Status changed from Assigned to Resolved
% Done changed from 60 to 100

It is up and everything looks fine.

Also available in: Atom PDF

Project

General

Profile

DCSO » Storage » CMS dCache

Issues

Custom queries