Project

General

Profile

Bug #10984

Upgrade FW on dCache-tape pools with qlcnic NIC

Added by Gerard Bernabeu Altayo almost 4 years ago. Updated almost 4 years ago.

Status:
Resolved
Priority:
Normal
Start date:
11/23/2015
Due date:
% Done:

0%

Estimated time:
component:
base
First Occurred:
Occurs In:
Stakeholders:
Co-Assignees:
Duration:

Description

Like I just did with the EOS FSTs, we need to upgrade the FW on those nodes that do not have it upgraded:

[root@cmsdcacheadmin ~]# curl http://localhost:2288/queueInfo | grep cell | grep E '(disk|tape)-disk' | sed 's/<td class="cell">//g' | sed 's/<\/td>//g' | cut -d f2 | sort -u
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 44919 0 44919 0 0 11.9M 0 -
:--:-- --:--:-- --:--:-- 21.4M
cmsstor261
cmsstor262
cmsstor264
cmsstor265
cmsstor266
cmsstor267
cmsstor268
cmsstor269
cmsstor270
cmsstor271
cmsstor272
cmsstor273
cmsstor274
cmsstor275
cmsstor276
cmsstor277
cmsstor278
cmsstor279

Reference in https://cdcvs.fnal.gov/redmine/issues/10982

One of this servers was upgraded first (cmsstor266).

History

#1 Updated by Chih-Hao Huang almost 4 years ago

  • Assignee changed from Gerard Bernabeu Altayo to Chih-Hao Huang

#2 Updated by Gerard Bernabeu Altayo almost 4 years ago

  • Assignee changed from Chih-Hao Huang to Gerard Bernabeu Altayo

Will do it on this list of nodes:

cmsstor261
cmsstor262
cmsstor264
cmsstor265
cmsstor267
cmsstor268
cmsstor269
cmsstor270
cmsstor271
cmsstor272
cmsstor273
cmsstor274
cmsstor275
cmsstor276
cmsstor277
cmsstor278
cmsstor279

Installing qlcnic in this nodes:

-bash-4.1$ pssh -l root -p50 --hosts=stor.tape.list -o pool-qlcnic.out -e pool-qlcnic.err -t 0 'rpm -q QConvergeConsoleCLI && exit; mkdir /root/qlcnic; cd /root/qlcnic; scp cmsstor266:/root/qlcnic/CLI/QConvergeConsoleCLI-1.1.04-65.x86_64.rpm . && rpm -vUh QConvergeConsoleCLI-1.1.04-65.x86_64.rpm;'

Now checking for the right location of the NIC:

-bash-4.1$ pssh -l root -p50 --hosts=stor.tape.list -o pool-qlcnic.out -e pool-qlcnic.err -t 0 'echo -e "1\n1\n1\n\n99\n" | /opt/QLogic_Corporation/QConvergeConsoleCLI/qaucli'
[1] 15:32:28 [FAILURE] cmsstor265 Exited with error code 253
[2] 15:32:28 [FAILURE] cmsstor270 Exited with error code 253
[3] 15:32:28 [FAILURE] cmsstor274 Exited with error code 253
[4] 15:32:28 [FAILURE] cmsstor268 Exited with error code 253
[5] 15:32:28 [FAILURE] cmsstor276 Exited with error code 253
[6] 15:32:28 [FAILURE] cmsstor269 Exited with error code 253
[7] 15:32:28 [FAILURE] cmsstor273 Exited with error code 253
[8] 15:32:28 [FAILURE] cmsstor264 Exited with error code 253
[9] 15:32:28 [FAILURE] cmsstor261 Exited with error code 253
[10] 15:32:28 [FAILURE] cmsstor279 Exited with error code 253
[11] 15:32:28 [FAILURE] cmsstor262 Exited with error code 253
[12] 15:32:28 [FAILURE] cmsstor278 Exited with error code 253
[13] 15:32:28 [FAILURE] cmsstor277 Exited with error code 253
[14] 15:32:28 [FAILURE] cmsstor272 Exited with error code 253
[15] 15:32:28 [FAILURE] cmsstor275 Exited with error code 253
[16] 15:32:28 [FAILURE] cmsstor271 Exited with error code 253
[17] 15:32:30 [FAILURE] cmsstor267 Exited with error code 253
-bash-4.1$ 
-bash-4.1$ cd pool-qlcnic.out/
-bash-4.1$ grep QLE32 cmsstor* | cut -d. -f1 | sort -u
cmsstor261:     1: CNA Model: QLE3240  SN: RFE1124D06019
cmsstor262:     1: CNA Model: QLE3240  SN: RFE1124D06015
cmsstor264:     1: CNA Model: QLE3240  SN: RFE1124D06047
cmsstor265:     1: CNA Model: QLE3240  SN: RFE1124D05918
cmsstor267:     1: CNA Model: QLE3240  SN: RFE1124D06043
cmsstor268:     1: CNA Model: QLE3240  SN: RFE1124D06416
cmsstor269:     1: CNA Model: QLE3240  SN: RFE1124D06048
cmsstor270:     1: CNA Model: QLE3240  SN: RFE1124D05869
cmsstor271:     1: CNA Model: QLE3240  SN: RFE1124D05946
cmsstor272:     1: CNA Model: QLE3240  SN: RFE1124D05975
cmsstor273:     1: CNA Model: QLE3240  SN: RFE1124D05889
cmsstor274:     1: CNA Model: QLE3240  SN: RFE1124D05858
cmsstor275:     1: CNA Model: QLE3240  SN: RFE1124D06026
cmsstor276:     1: CNA Model: QLE3240  SN: RFE1124D06056
cmsstor277:     1: CNA Model: QLE3240  SN: RFE1124D05942
cmsstor278:     1: CNA Model: QLE3240  SN: RFE1124D05990
cmsstor279:     1: CNA Model: QLE3240  SN: RFE1124D06018
-bash-4.1$ grep QLE32 cmsstor* | cut -d. -f1 | sort -u | wc -l
17
-bash-4.1$ cd ..
-bash-4.1$ wc -l stor.tape.list 
17 stor.tape.list
-bash-4.1$ 

Since its Monday and we still have 3 days to see if they've issues, I'll perform the upgrade...

-bash-4.1$  pssh -l root -p50 --hosts=stor.tape.list -o pool-qlcnic.out -e pool-qlcnic.err -t 0 'ethtool -i eth0'
[1] 15:34:34 [SUCCESS] cmsstor262
[2] 15:34:34 [SUCCESS] cmsstor264
[3] 15:34:34 [SUCCESS] cmsstor261
[4] 15:34:34 [SUCCESS] cmsstor265
[5] 15:34:34 [SUCCESS] cmsstor267
[6] 15:34:34 [SUCCESS] cmsstor271
[7] 15:34:34 [SUCCESS] cmsstor268
[8] 15:34:34 [SUCCESS] cmsstor270
[9] 15:34:34 [SUCCESS] cmsstor269
[10] 15:34:34 [SUCCESS] cmsstor273
[11] 15:34:34 [SUCCESS] cmsstor277
[12] 15:34:34 [SUCCESS] cmsstor276
[13] 15:34:34 [SUCCESS] cmsstor278
[14] 15:34:34 [SUCCESS] cmsstor274
[15] 15:34:34 [SUCCESS] cmsstor279
[16] 15:34:34 [SUCCESS] cmsstor272
[17] 15:34:34 [SUCCESS] cmsstor275
-bash-4.1$ grep firmw pool-qlcnic.out/cmsstor*
pool-qlcnic.out/cmsstor261:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor262:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor264:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor265:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor267:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor268:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor269:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor270:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor271:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor272:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor273:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor274:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor275:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor276:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor277:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor278:firmware-version: 4.12.10
pool-qlcnic.out/cmsstor279:firmware-version: 4.12.10
-bash-4.1$ 

#3 Updated by Gerard Bernabeu Altayo almost 4 years ago

Performing the actual update:

-bash-4.1$  pssh -l root -p50 --hosts=stor.tape.list -o pool-qlcnic.out -e pool-qlcnic.err -t 0 'scp cmsstor266:/root/qlcnic/p3p23204.bin /root/qlcnic/p3p23204.bin; lsmod | grep qlcnic && lspci | grep cLOM8214 && echo -e "3\n1\n1\n1\n/root/qlcnic/p3p23204.bin\nyes\nyes\n\n99\n" | /opt/QLogic_Corporation/QConvergeConsoleCLI/qaucli'
[1] 15:36:13 [FAILURE] cmsstor279 Exited with error code 253
[2] 15:36:13 [FAILURE] cmsstor277 Exited with error code 253
[3] 15:39:25 [FAILURE] cmsstor276 Exited with error code 253
[4] 15:39:26 [FAILURE] cmsstor262 Exited with error code 253
[5] 15:39:26 [FAILURE] cmsstor274 Exited with error code 253
[6] 15:39:26 [FAILURE] cmsstor278 Exited with error code 253
[7] 15:39:27 [FAILURE] cmsstor272 Exited with error code 253
[8] 15:39:27 [FAILURE] cmsstor265 Exited with error code 253
[9] 15:39:27 [FAILURE] cmsstor271 Exited with error code 253
[10] 15:39:29 [FAILURE] cmsstor268 Exited with error code 253
[11] 15:39:30 [FAILURE] cmsstor270 Exited with error code 253
[12] 15:39:30 [FAILURE] cmsstor273 Exited with error code 253
[13] 15:39:30 [FAILURE] cmsstor269 Exited with error code 253
[14] 15:39:32 [FAILURE] cmsstor275 Exited with error code 253
[15] 15:39:32 [FAILURE] cmsstor261 Exited with error code 253
[16] 15:39:33 [FAILURE] cmsstor267 Exited with error code 253
[17] 15:39:43 [FAILURE] cmsstor264 Exited with error code 253
-bash-4.1$  pssh -l root -p50 --hosts=stor.tape.list -o pool-qlcnic.out -e pool-qlcnic.err -t 0 'ethtool -i eth0'
[1] 15:42:03 [SUCCESS] cmsstor262
[2] 15:42:03 [SUCCESS] cmsstor264
[3] 15:42:03 [SUCCESS] cmsstor267
[4] 15:42:03 [SUCCESS] cmsstor268
[5] 15:42:03 [SUCCESS] cmsstor270
[6] 15:42:03 [SUCCESS] cmsstor277
[7] 15:42:03 [SUCCESS] cmsstor269
[8] 15:42:03 [SUCCESS] cmsstor274
[9] 15:42:03 [SUCCESS] cmsstor278
[10] 15:42:03 [SUCCESS] cmsstor265
[11] 15:42:03 [SUCCESS] cmsstor279
[12] 15:42:03 [SUCCESS] cmsstor276
[13] 15:42:03 [SUCCESS] cmsstor272
[14] 15:42:03 [SUCCESS] cmsstor273
[15] 15:42:03 [SUCCESS] cmsstor261
[16] 15:42:03 [SUCCESS] cmsstor275
[17] 15:42:04 [SUCCESS] cmsstor271
-bash-4.1$ grep firmw pool-qlcnic.out/cmsstor*
pool-qlcnic.out/cmsstor261:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor262:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor264:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor265:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor267:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor268:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor269:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor270:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor271:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor272:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor273:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor274:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor275:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor276:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor277:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor278:firmware-version: 4.20.1
pool-qlcnic.out/cmsstor279:firmware-version: 4.20.1
-bash-4.1$ 

It's all successful, I will close this ticket and open a new one to perform this on ALL the CMS HW inventory. The most important pending servers are the dCache-disk pools and many servers that are on old FW revisions of this HW (on some servers this is causing packet loss!!!):

https://ecfmon1.fnal.gov/dcsomon/check_mk/index.py?start_url=%2Fdcsomon%2Fcheck_mk%2Fview.py%3Fview_name%3Dhoststatus%26host%3Dcmschimeradiskbackup

#4 Updated by Gerard Bernabeu Altayo almost 4 years ago

  • Subject changed from Upgrade FW on dCache pools with qlcnic NIC to Upgrade FW on dCache-tape pools with qlcnic NIC
  • Status changed from New to Resolved


Also available in: Atom PDF