Project

General

Profile

Task #9784

Prepare for downtime Aug 5th

Added by Natalia Ratnikova over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
High
Start date:
08/04/2015
Due date:
% Done:

0%

Estimated time:
Spent time:
Duration:

Description

get list of commands to execute during the downtime on dCache disk and phedex servers.
Test upgrades:
java and kernel upgrade , qlcnic firmware upgrade on dcache pool servers

History

#1 Updated by Natalia Ratnikova over 4 years ago

cmsstor112 was ust reinstalled as dcache disk_itb pool and should get all the updates .

[root@cmsstor112 ~]# java -version
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
[root@cmsstor112 ~]# rpm -qf `which java`
file /usr/bin/java is not owned by any package
[root@cmsstor112 ~]# ls -l /usr/bin/java
lrwxrwxrwx. 1 root root 26 Aug 4 11:18 /usr/bin/java -> /usr/java/default/bin/java
[root@cmsstor112 ~]# ls -l /usr/java/default/bin/java
-rwxr-xr-x. 1 root root 7750 Jun 5 2013 /usr/java/default/bin/java
[root@cmsstor112 ~]# rpm -qf /usr/java/default/bin/java
jdk-1.7.0_25-fcs.x86_64
[root@cmsstor112 ~]# facter kernelrelease
2.6.32-504.30.3.el6.x86_64

#2 Updated by Natalia Ratnikova over 4 years ago

[root@cmsstor143 ~]# java -version
java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)
[root@cmsstor143 ~]# uptime
11:57:12 up 173 days, 21:56, 1 user, load average: 0.08, 0.07, 0.06
[root@cmsstor143 ~]# facter kernelrelease
2.6.32-504.8.1.el6.x86_64
[root@cmsstor143 ~]#

#3 Updated by Natalia Ratnikova over 4 years ago

Hi Yujun,

unfortunately Dmitry is on vacation.

Would it be possible that you or other FNAL dCache admins give usthe information on java version compatible with dCache 2.2,
and/or what is currently used for public dCache instance?
We would like to try it on our test instance before tomorrow's down time.

Thanks,
Natalia.

-------- Forwarded Message --------
Message-ID: <>
Date: Tue, 04 Aug 2015 14:56:47 -0500
From: Natalia Ratnikova <>
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:31.0) Gecko/20100101 Thunderbird/31.7.0
MIME-Version: 1.0
To: Dmitry Litvintsev <>, dcso <>
Subject: java version for dCache
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Dmitry,

our cms-t1 dCache instances are currently using a pretty old frozen
version of java, [1] below. We'd like to move onto a new version of
openjdk from slf repository, for security updates, and ask for your
recommendation:

- which new version would you recommend us to use?
- any known constraints specific to dCache version 2.2?
- what set of packages do we need to install?

The [2] below contains a list of openjdk packages/version we have
available from already enabled repositories.
Perhaps we could just install what is known to work on the public
dCache instance?

Thanks,
Natalia.

[1]
[root@cmsstor373 ~]# rpm -qi `rpm -qf /usr/java/default/bin/java`
Name : jdk Relocations: /usr/java
Version : 1.7.0_25 Vendor: Sun
Microsystems, Inc.
Release : fcs Build Date: Wed 05 Jun 2013
11:09:47 PM CDT
Install Date: Wed 19 Mar 2014 05:01:24 PM CDT Build Host:
sc11137388.us.oracle.com
Group : Development/Tools Source RPM:
jdk-1.7.0_25-fcs.src.rpm
Size : 156918506 License:
http://java.com/license
Signature : (none)
Packager : Java Software <>
URL : URL_REF
Summary : Java Platform Standard Edition Development Kit
Description :
The Java Platform Standard Edition Development Kit (JDK) includes both
the runtime environment (Java virtual machine, the Java platform classes
and supporting files) and development tools (compilers, debuggers,
tool libraries and other tools).

The JDK is a development environment for building applications, applets
and components that can be deployed with the Java Platform Standard
Edition Runtime Environment.

[2]

[root@cmsstor373 ~]# yum list | grep openjdk
java-1.6.0-openjdk.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-demo.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-devel.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-javadoc.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-src.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.7.0-openjdk.x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.7.0-openjdk-demo.x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.7.0-openjdk-devel.x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.7.0-openjdk-javadoc.noarch 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.7.0-openjdk-src.x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.8.0-openjdk.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-demo.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-devel.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-javadoc.noarch 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-src.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
osg-java7-compat-openjdk.noarch 1.0-1.osg32.el6 osg
osg-java7-devel-compat-openjdk.noarch 1.0-1.osg32.el6 osg

#4 Updated by Natalia Ratnikova over 4 years ago

Currently available puppet implementations for java install:

[cmsdev33 15:59] git blame ./modules/osg_gatekeeper/manifests/install.pp
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 1)
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 2) class osg_gatekeeper::install {
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 3) # We have a situation where we require java 7, but left on it's own, yum
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 4) # picks java 6. Additionally, the uscmst1 repository, which has a higher priority
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 5) # than all other repositories, contains the Oracle Java 7. We do not want this.
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 6) # So, we disable the OSG and uscmst1 repositories and specifically install the
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 7) # OpenJDK version of Java 7. It is a nasty hack and hopefully in the next few
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 8) # iterations, it will no longer be necessary.
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 9) exec { 'install_java_7':
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 10) command => 'yum --disablerepo=osg --disablerepo=uscmst1 -y install java-1.7.0-openjdk',
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 11) unless => 'rpm -q java-1.7.0-openjdk',
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 12) before => Package['osg-ce-condor'],
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 13) }
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 14) package { 'osg-ce-condor':
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 15) ensure => present,
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 16) }
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 17) package { 'osg-control':
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 18) ensure => present,
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 19) require => [ Package['osg-ce-condor'], ]
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 20) }
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 21) package { 'uscms-fts-provider':
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 22) ensure => present,
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 23) require => Package['osg-ce-condor'],
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 24) }
0e0e1c92 (Anthony Tiradani 2014-03-21 16:19:06 -0500 25) }
[cmsdev33 15:59] grep jdk `find . -type f `
Binary file ./.git/objects/pack/pack-48fb44b376b22a0355e6d97002c83584f312fbda.pack matches
Binary file ./.git/objects/pack/pack-48fb44b376b22a0355e6d97002c83584f312fbda.idx matches
./modules/osg_gatekeeper/manifests/install.pp: command => 'yum --disablerepo=osg --disablerepo=uscmst1 -y install java-1.7.0-openjdk',
./modules/osg_gatekeeper/manifests/install.pp: unless => 'rpm -q java-1.7.0-openjdk',
./modules/dcache/manifests/init.pp: ensure_packages ( ['dcache', 'jdk', 'rpcbind', 'nfs-utils', $packages] )
./modules/dcache/manifests/init.pp: require => [ Package['dcache'], Package['jdk'] ]

#5 Updated by Natalia Ratnikova over 4 years ago

Quote from https://www.dcache.org/manuals/upgrade/upgrade-2.2-to-2.6.html :

Miscellaneous
Java 7

Java 7 is now a required dependency for dCache. As always, we recommend installing the JDK over the JRE, because the JDK contains additional debugging tools.

Commits: 5c18a89.

#6 Updated by Natalia Ratnikova over 4 years ago

On 8/4/15 4:08 PM, Yujun Wu wrote:

Hi Natalia,

The version used on public FNDCA dcache pool nodes (SL5) is:
java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el5_10.x86_64
( [root@stkendca42a ~]# rpm -qa|grep java
java-1.7.0-openjdk-1.7.0.45-2.4.3.1.el5_10.x86_64
)

Our dcache header nodes (SL6) are using a similar version
java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64
([root@fndca4a ~]# rpm -qa|grep java
tzdata-java-2014h-1.el6.noarch
java-1.7.0-openjdk-1.7.0.71-2.5.3.1.el6.x86_64
java-1.7.0-openjdk-devel-1.7.0.71-2.5.3.1.el6.x86_64
)

Right now, FNDCA Public dCache nodes can pull the openjdk java from yum. You can also get it directory from Fermilab linux site. I have an elog for this:

http://www-enstore.fnal.gov:8181/dCache/526

(you may want to pick up the newer version than the one mentioned in the elog at that time).

The only other thing we found is java.security need update a little bit. Please see the elog as well.

Also, fndca is running dCache 2.6.34. I remember the openjdk java 1.7 works with 2.2.

Please just let me know if you have any issues. I am cc-ing to dcache-admin mailing list. It has everybody (developers plus admin) on the list. Other people can reply back and/or correct if needed.

Regards,
Yujun

#7 Updated by Natalia Ratnikova over 4 years ago

Install openjdk on cmsstor112, and start dcache.

cmsstor112.fnal.gov - dcachepooldisk_itb/production (SLF 6.6)
4-core Core Opteron 265 (H8DSP-8); 7.63 GB RAM, 32.00 GB swap
[root@cmsstor112 ~]# rpm q java-1.7.0-openjdk
package java-1.7.0-openjdk is not installed
[root@cmsstor112 ~]# echo $?
1
[root@cmsstor112 ~]# dcache status
DOMAIN STATUS PID USER
w-cmsstor112-disk_itb-disk1Domain running 8333 root
w-cmsstor112-disk_itb-disk2Domain running 8385 root
w-cmsstor112-disk_itb-disk3Domain running 8436 root
gridftp-cmsstor112Domain running 8490 root
[root@cmsstor112 ~]# service puppet status
puppet (pid 6142) is running...
[root@cmsstor112 ~]# service puppet stop
Stopping puppet agent: [ OK ]
[root@cmsstor112 ~]# yum list jdk
Loaded plugins: priorities, security
7900 packages excluded due to repository priority protections
Installed Packages
jdk.x86_64 2000:1.7.0_25-fcs @uscmst1
[root@cmsstor112 ~]# yum erase jdk
Loaded plugins: priorities, security
Setting up Remove Process
Resolving Dependencies
-
> Running transaction check
---> Package jdk.x86_64 2000:1.7.0_25-fcs will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size ==========================================================================================================================================================================================================================================================================================================================================================================
Removing:
jdk x86_64 2000:1.7.0_25-fcs @uscmst1 150 M

Transaction Summary ==========================================================================================================================================================================================================================================================================================================================================================================
Remove 1 Package(s)

Installed size: 150 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Erasing : 2000:jdk-1.7.0_25-fcs.x86_64 1/1
Verifying : 2000:jdk-1.7.0_25-fcs.x86_64 1/1

Removed:
jdk.x86_64 2000:1.7.0_25-fcs

Complete!
[root@cmsstor112 ~]# dcache status
which: no java in (/usr/krb5/sbin:/usr/krb5/bin:/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin)
Could not find usable Java VM. Please set JAVA_HOME to the path to Java 6
or newer.
[root@cmsstor112 ~]# service dcache status
dcache: unrecognized service
[root@cmsstor112 ~]# service dcache-server status
which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)
Could not find usable Java VM. Please set JAVA_HOME to the path to Java 6
or newer.
[root@cmsstor112 ~]# service dcache-server stop
which: no java in (/sbin:/usr/sbin:/bin:/usr/bin)
Could not find usable Java VM. Please set JAVA_HOME to the path to Java 6
or newer.
[root@cmsstor112 ~]# yum install jdk
Loaded plugins: priorities, security
Setting up Install Process
7900 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package jdk.x86_64 2000:1.7.0_25-fcs will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size ==========================================================================================================================================================================================================================================================================================================================================================================
Installing:
jdk x86_64 2000:1.7.0_25-fcs uscmst1 81 M

Transaction Summary ==========================================================================================================================================================================================================================================================================================================================================================================
Install 1 Package(s)

Total download size: 81 M
Installed size: 150 M
Is this ok [y/N]: y
Downloading Packages:
oracle-jdk-1.7.0_25-fcs.x86_64.rpm | 81 MB 00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : 2000:jdk-1.7.0_25-fcs.x86_64 1/1
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
Verifying : 2000:jdk-1.7.0_25-fcs.x86_64 1/1

Installed:
jdk.x86_64 2000:1.7.0_25-fcs

Complete!
[root@cmsstor112 ~]# service dcache-server stop
Stopping gridftp-cmsstor112Domain 0 done
Stopping w-cmsstor112-disk_itb-disk3Domain 0 1 done
Stopping w-cmsstor112-disk_itb-disk2Domain 0 1 done
Stopping w-cmsstor112-disk_itb-disk1Domain 0 1 done
[root@cmsstor112 ~]# dcache status
DOMAIN STATUS PID USER
w-cmsstor112-disk_itb-disk1Domain stopped root
w-cmsstor112-disk_itb-disk2Domain stopped root
w-cmsstor112-disk_itb-disk3Domain stopped root
gridftp-cmsstor112Domain stopped root
[root@cmsstor112 ~]# yum erase jdk
Loaded plugins: priorities, security
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package jdk.x86_64 2000:1.7.0_25-fcs will be erased
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size ==========================================================================================================================================================================================================================================================================================================================================================================
Removing:
jdk x86_64 2000:1.7.0_25-fcs @uscmst1 150 M

Transaction Summary ==========================================================================================================================================================================================================================================================================================================================================================================
Remove 1 Package(s)

Installed size: 150 M
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Erasing : 2000:jdk-1.7.0_25-fcs.x86_64 1/1
Verifying : 2000:jdk-1.7.0_25-fcs.x86_64 1/1

Removed:
jdk.x86_64 2000:1.7.0_25-fcs

Complete!
[root@cmsstor112 ~]# yum --disablerepo=osg --disablerepo=uscmst1 y install java-1.7.0-openjdk
Loaded plugins: priorities, security
Setting up Install Process
7536 packages excluded due to repository priority protections
Resolving Dependencies
-
> Running transaction check
---> Package java-1.7.0-openjdk.x86_64 1:1.7.0.85-2.6.1.3.el6_6 will be installed
--> Processing Dependency: jpackage-utils >= 1.7.3-1jpp.2 for package: 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
--> Processing Dependency: xorg-x11-fonts-Type1 for package: 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
--> Processing Dependency: tzdata-java for package: 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
--> Processing Dependency: libpulse.so.0(PULSE_0)(64bit) for package: 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
--> Processing Dependency: libpulse.so.0()(64bit) for package: 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
--> Processing Dependency: libpcsclite.so.1()(64bit) for package: 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
--> Running transaction check
---> Package jpackage-utils.noarch 0:1.7.5-3.12.el6 will be installed
---> Package pcsc-lite-libs.x86_64 0:1.5.2-14.el6 will be installed
---> Package pulseaudio-libs.x86_64 0:0.9.21-17.el6 will be installed
--> Processing Dependency: libsndfile.so.1(libsndfile.so.1.0)(64bit) for package: pulseaudio-libs-0.9.21-17.el6.x86_64
--> Processing Dependency: libsndfile.so.1()(64bit) for package: pulseaudio-libs-0.9.21-17.el6.x86_64
--> Processing Dependency: libasyncns.so.0()(64bit) for package: pulseaudio-libs-0.9.21-17.el6.x86_64
---> Package tzdata-java.noarch 0:2015d-1.el6 will be installed
---> Package xorg-x11-fonts-Type1.noarch 0:7.2-9.1.el6 will be installed
--> Processing Dependency: ttmkfdir for package: xorg-x11-fonts-Type1-7.2-9.1.el6.noarch
--> Processing Dependency: ttmkfdir for package: xorg-x11-fonts-Type1-7.2-9.1.el6.noarch
--> Running transaction check
---> Package libasyncns.x86_64 0:0.8-1.1.el6 will be installed
---> Package libsndfile.x86_64 0:1.0.20-5.el6 will be installed
--> Processing Dependency: libFLAC.so.8()(64bit) for package: libsndfile-1.0.20-5.el6.x86_64
---> Package ttmkfdir.x86_64 0:3.0.9-32.1.el6 will be installed
--> Running transaction check
---> Package flac.x86_64 0:1.2.1-7.el6_6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==========================================================================================================================================================================================================================================================================================================================================================================
Package Arch Version Repository Size ==========================================================================================================================================================================================================================================================================================================================================================================
Installing:
java-1.7.0-openjdk x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security 26 M
Installing for dependencies:
flac x86_64 1.2.1-7.el6_6 slf-security 242 k
jpackage-utils noarch 1.7.5-3.12.el6 slf 58 k
libasyncns x86_64 0.8-1.1.el6 slf 23 k
libsndfile x86_64 1.0.20-5.el6 slf 232 k
pcsc-lite-libs x86_64 1.5.2-14.el6 slf 27 k
pulseaudio-libs x86_64 0.9.21-17.el6 slf 461 k
ttmkfdir x86_64 3.0.9-32.1.el6 slf 42 k
tzdata-java noarch 2015d-1.el6 slf-security 174 k
xorg-x11-fonts-Type1 noarch 7.2-9.1.el6 slf 519 k

Transaction Summary ==========================================================================================================================================================================================================================================================================================================================================================================
Install 10 Package(s)

Total download size: 28 M
Installed size: 96 M
Downloading Packages:
(1/10): flac-1.2.1-7.el6_6.x86_64.rpm | 242 kB 00:00
(2/10): java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64.rpm | 26 MB 00:01
(3/10): jpackage-utils-1.7.5-3.12.el6.noarch.rpm | 58 kB 00:00
(4/10): libasyncns-0.8-1.1.el6.x86_64.rpm | 23 kB 00:00
(5/10): libsndfile-1.0.20-5.el6.x86_64.rpm | 232 kB 00:00
(6/10): pcsc-lite-libs-1.5.2-14.el6.x86_64.rpm | 27 kB 00:00
(7/10): pulseaudio-libs-0.9.21-17.el6.x86_64.rpm | 461 kB 00:00
(8/10): ttmkfdir-3.0.9-32.1.el6.x86_64.rpm | 42 kB 00:00
(9/10): tzdata-java-2015d-1.el6.noarch.rpm | 174 kB 00:00
(10/10): xorg-x11-fonts-Type1-7.2-9.1.el6.noarch.rpm | 519 kB 00:00
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total 18 MB/s | 28 MB 00:01
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing : jpackage-utils-1.7.5-3.12.el6.noarch 1/10
Installing : libasyncns-0.8-1.1.el6.x86_64 2/10
Installing : tzdata-java-2015d-1.el6.noarch 3/10
Installing : pcsc-lite-libs-1.5.2-14.el6.x86_64 4/10
Installing : flac-1.2.1-7.el6_6.x86_64 5/10
Installing : libsndfile-1.0.20-5.el6.x86_64 6/10
Installing : pulseaudio-libs-0.9.21-17.el6.x86_64 7/10
Installing : ttmkfdir-3.0.9-32.1.el6.x86_64 8/10
Installing : xorg-x11-fonts-Type1-7.2-9.1.el6.noarch 9/10
Installing : 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64 10/10
Verifying : ttmkfdir-3.0.9-32.1.el6.x86_64 1/10
Verifying : xorg-x11-fonts-Type1-7.2-9.1.el6.noarch 2/10
Verifying : flac-1.2.1-7.el6_6.x86_64 3/10
Verifying : pcsc-lite-libs-1.5.2-14.el6.x86_64 4/10
Verifying : tzdata-java-2015d-1.el6.noarch 5/10
Verifying : libasyncns-0.8-1.1.el6.x86_64 6/10
Verifying : 1:java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64 7/10
Verifying : pulseaudio-libs-0.9.21-17.el6.x86_64 8/10
Verifying : libsndfile-1.0.20-5.el6.x86_64 9/10
Verifying : jpackage-utils-1.7.5-3.12.el6.noarch 10/10

Installed:
java-1.7.0-openjdk.x86_64 1:1.7.0.85-2.6.1.3.el6_6

Dependency Installed:
flac.x86_64 0:1.2.1-7.el6_6 jpackage-utils.noarch 0:1.7.5-3.12.el6 libasyncns.x86_64 0:0.8-1.1.el6 libsndfile.x86_64 0:1.0.20-5.el6 pcsc-lite-libs.x86_64 0:1.5.2-14.el6 pulseaudio-libs.x86_64 0:0.9.21-17.el6 ttmkfdir.x86_64 0:3.0.9-32.1.el6 tzdata-java.noarch 0:2015d-1.el6
xorg-x11-fonts-Type1.noarch 0:7.2-9.1.el6

Complete!
[root@cmsstor112 ~]# yum list | grep jdk
java-1.7.0-openjdk.x86_64 1:1.7.0.85-2.6.1.3.el6_6 @slf-security
java-1.6.0-openjdk.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-demo.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-devel.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-javadoc.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.6.0-openjdk-src.x86_64 1:1.6.0.35-1.13.7.1.el6_6
java-1.7.0-openjdk-demo.x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.7.0-openjdk-devel.x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.7.0-openjdk-javadoc.noarch 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.7.0-openjdk-src.x86_64 1:1.7.0.85-2.6.1.3.el6_6 slf-security
java-1.8.0-openjdk.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-demo.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-devel.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-javadoc.noarch 1:1.8.0.51-0.b16.el6_6 slf-security
java-1.8.0-openjdk-src.x86_64 1:1.8.0.51-0.b16.el6_6 slf-security
jdk.x86_64 2000:1.7.0_25-fcs uscmst1
ldapjdk.x86_64 4.18-6.el6 slf
ldapjdk-javadoc.x86_64 4.18-6.el6 slf
maven30-slf4j-jdk14.noarch 1.7.4-3.11.el6 slf-collections
osg-java7-compat-openjdk.noarch 1.0-1.osg32.el6 osg
osg-java7-devel-compat-openjdk.noarch 1.0-1.osg32.el6 osg
[root@cmsstor112 ~]# whcih java
-bash: whcih: command not found
[root@cmsstor112 ~]# which java
/usr/bin/java
[root@cmsstor112 ~]# java -version
java version "1.7.0_85"
OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)
OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)
[root@cmsstor112 ~]# locate java.security.orig
(reverse-i-search)`': ls /usr/lib/jvm/java-1.7.0-openjdk^C
[root@cmsstor112 ~]# ls /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/
cacerts java.policy java.security local_policy.jar nss.cfg US_export_policy.jar
[root@cmsstor112 ~]# wc -l /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
501 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
[root@cmsstor112 ~]# grep =MD2 /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
  1. jdk.certpath.disabledAlgorithms=MD2, DSA, RSA keySize < 2048
    jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024
    [root@cmsstor112 ~]# nano w /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
    [root@cmsstor112 ~]# cp -p /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security.orig
    [root@cmsstor112 ~]# nano -w /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
    [root@cmsstor112 ~]# diff /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security.orig
    416c416
    < jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024 # NR
    --

jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024

[root@cmsstor112 ~]# mv /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security.orig /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
mv: overwrite `/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security'? y
[root@cmsstor112 ~]# rpm -qf /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
[root@cmsstor112 ~]# rpm -V java-1.7.0-openjdk-1.7.0.85-2.6.1.3.el6_6.x86_64
[root@cmsstor112 ~]# dcache status
DOMAIN STATUS PID USER
w-cmsstor112-disk_itb-disk1Domain stopped root
w-cmsstor112-disk_itb-disk2Domain stopped root
w-cmsstor112-disk_itb-disk3Domain stopped root
gridftp-cmsstor112Domain stopped root
[root@cmsstor112 ~]# dcache start
Starting w-cmsstor112-disk_itb-disk1Domain done
Starting w-cmsstor112-disk_itb-disk2Domain done
Starting w-cmsstor112-disk_itb-disk3Domain done
Starting gridftp-cmsstor112Domain done
[root@cmsstor112 ~]# dcache status
DOMAIN STATUS PID USER
w-cmsstor112-disk_itb-disk1Domain running 7137 root
w-cmsstor112-disk_itb-disk2Domain running 7191 root
w-cmsstor112-disk_itb-disk3Domain running 7252 root
gridftp-cmsstor112Domain running 7305 root
[root@cmsstor112 ~]#

#8 Updated by Natalia Ratnikova over 4 years ago

make_branch natalia_dcache_java_upgrade_test

dcache_irtb instance head node is currently using natalia_burning_dcache_pools branch

To get all recent changes:

merge itb branch into natalia_burning_dcache_pools branch

merge natalia_burning_dcache_pools into the newly created natalia_dcache_java_upgrade_test branch

Now edit pool manager configuration:

[cmsdev33 15:59] git show
commit 853bea9504df2e85758e2a9fc8de863d00d8d1a9
Author: Natalia Ratnikova <>
Date: Tue Aug 4 17:33:07 2015 -0500

Added cmsstor112 and start burning tests there.

diff --git a/modules/dcache/files/etc/dcache/poolmanager-disk_itb.conf b/modules/dcache/files/etc/dcache/poolmanager-disk_itb.conf
index 5a99360..d81a24b 100644
--- a/modules/dcache/files/etc/dcache/poolmanager-disk_itb.conf
+++ b/modules/dcache/files/etc/dcache/poolmanager-disk_itb.conf
@ -36,6 +36,9 @ psu addto ugroup any-protocol */* # The pools ... #

+psu create pool w-cmsstor112-disk_itb-disk1
+psu create pool w-cmsstor112-disk_itb-disk2
+psu create pool w-cmsstor112-disk_itb-disk3
psu create pool w-cmsstor151-disk_itb-disk1
psu create pool w-cmsstor151-disk_itb-disk2
psu create pool w-cmsstor151-disk_itb-disk3
@ -51,6 +54,10 @ psu create pgroup burningPools
psu create pgroup readonlyPools

psu addto pgroup burningPools w-cmsstor112-disk_itb-disk1
+psu addto pgroup burningPools w-cmsstor112-disk_itb-disk2
+psu addto pgroup burningPools w-cmsstor112-disk_itb-disk3

psu addto pgroup flushPools w-cmsstor155-disk_itb-disk1
psu addto pgroup flushPools w-cmsstor155-disk_itb-disk2
psu addto pgroup flushPools w-cmsstor155-disk_itb-disk3
[cmsdev33 15:59] git push

In the ENC : move onto a new environment for java upgrade test.

[cmsdev33 15:59] git diff r 38fc40812144fce7e080fbb19f1f7fa40201e5fb
diff --git a/hosts/cmsstor152.fnal.gov.yaml b/hosts/cmsstor152.fnal.gov.yaml
index 744d19b..95420d7 100644
--
a/hosts/cmsstor152.fnal.gov.yaml
+++ b/hosts/cmsstor152.fnal.gov.yaml
@ -1,8 +1,4 @
classes:
role::storage::dcacheadmindisk_itb:
- rpmrepos::epel:
- itb: 'true'
- rpmrepos::osg:
- itb: 'true'

-environment: natalia_burning_dcache_pools
+environment: natalia_dcache_java_upgrade_test
[cmsdev33 15:59] git status
On branch master
Your branch is up-to-date with 'origin/master'.

nothing to commit, working directory clean =============
To start burning, login to admin interface and start migration processes.

#9 Updated by Natalia Ratnikova over 4 years ago

For some reason burningPools and readonlyPools groups were not existing on the testbed, and not shown in monitoring pages for groups.

TO force the upadtes poolmangaer.conf to be propagated, change it again (remove some empty lines), commit, push , and rerun puppet on cmsstor152 - admin node . After that all three groups appeared in monitoring and in admin interface :

[root@cmsstor152 ~]# ssh -1 -i /etc/dcache/admin/server_key -c blowfish -p 22223 cmsstor152.fnal.gov -l admin

dCache Admin (VII) (user=admin)

[cmsstor152.fnal.gov] (local) admin > cd PoolManager
[cmsstor152.fnal.gov] (PoolManager) admin > psu ls pool
w-cmsstor155-disk_itb-disk1
w-cmsstor112-disk_itb-disk3
w-cmsstor112-disk_itb-disk1
w-cmsstor112-disk_itb-disk2
w-cmsstor151-disk_itb-disk2
w-cmsstor151-disk_itb-disk3
w-cmsstor155-disk_itb-disk2
w-cmsstor155-disk_itb-disk3
w-cmsstor151-disk_itb-disk1
[cmsstor152.fnal.gov] (PoolManager) admin > psu ls pgroup
flushPools
burningPools
readonlyPools

Migration is now running (before it could not find available targets):
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk1) admin > migration ls
[3] RUNNING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[2] SLEEPING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk1) admin > migration cancel 2
[2] CANCELLED migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk1) admin > migration clear
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk1) admin > migration ls
[3] RUNNING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools

It looks like migration process needs to be restarted, to get from sleeping to running state.

[2] SLEEPING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[1] FINISHED migration copy target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration clear
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration show
Syntax error: (2) Command not found : show
Help :
migration cache [options] <target> ...
migration cancel [-force] <job>
migration clear
migration concurrency <job> <n>
migration copy [options] <target> ...
migration info <job>
migration ls
migration move [options] <target> ...
migration resume <job>
migration suspend <job>

[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration info 1
java.util.NoSuchElementException: Job not found
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration info 2
Command : migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
State : SLEEPING
Queued : 0
Attempts : 67879
Targets : w-cmsstor112-disk_itb-disk3,w-cmsstor112-disk_itb-disk1,w-cmsstor112-disk_itb-disk2
Completed : 67879 files; 2150956345466 bytes; 100%
Total : 2150956345466 bytes
Concurrency: 10
Running tasks:
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration ls
[2] SLEEPING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration cancel 2
[2] CANCELLED migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration clear
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration ls
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup burningPools
[3] INITIALIZING migration copy -verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup -
burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > migration ls
[3] RUNNING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin >
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk2) admin > ..
[cmsstor152.fnal.gov] (local) admin > cd w-cmsstor155-disk_itb-disk3
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration ls
[2] SLEEPING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[1] FINISHED migration copy target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration clear
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration ls
[2] SLEEPING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration cancel 2
[2] CANCELLED migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration clear
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration ls
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup burningPools
[3] INITIALIZING migration copy -verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup -
burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration ls
[3] RUNNING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin >

Migration now works . See if it persists after dcache service restart : YES!

[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > ..
[cmsstor152.fnal.gov] (local) admin > cd w-cmsstor155-disk_itb-disk3
[cmsstor152.fnal.gov] (w-cmsstor155-disk_itb-disk3) admin > migration info 3
Command : migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
State : RUNNING
Queued : 667
Attempts : 10
Targets : w-cmsstor112-disk_itb-disk3,w-cmsstor112-disk_itb-disk1,w-cmsstor112-disk_itb-disk2
Completed : 0 files; 0 bytes; 0%
Total : 1835024777216 bytes
Concurrency: 10
Running tasks:
[135680] 0000AEEB892B582F40A59B070658ABDB8DA7: TASK.Copying -> [w-cmsstor112-disk_itb-disk3@local]
[135681] 0000D20DCD43CF454CD5AF24860D4816AD4F: TASK.Copying -> [w-cmsstor112-disk_itb-disk2@local]
[135682] 000093EA33521AB44CF897121EB661A3A530: TASK.Copying -> [w-cmsstor112-disk_itb-disk3@local]
[135683] 00007CDD8F1285DF4BB0A152040F05A525A8: TASK.Copying -> [w-cmsstor112-disk_itb-disk2@local]
[135684] 00002A0FEB9FE9554A23B8571A7C52E9BDE6: TASK.Copying -> [w-cmsstor112-disk_itb-disk1@local]
[135685] 00006EBC8FE4BDD1450BBC0D9F1CF2F57484: TASK.Copying -> [w-cmsstor112-disk_itb-disk3@local]
[135686] 000031D80690EB8248C389A0CC91D85B14E1: TASK.Copying -> [w-cmsstor112-disk_itb-disk3@local]
[135687] 00009A83EF6A069E40AF8BBA8643C9241777: TASK.Copying -> [w-cmsstor112-disk_itb-disk2@local]
[135688] 000089921355B6C7436A99FCC7439353B235: TASK.Copying -> [w-cmsstor112-disk_itb-disk1@local]
[135689] 000097A3B94B8FB547CEAEE47BD4649C0339: TASK.Copying -> [w-cmsstor112-disk_itb-disk1@local]

#10 Updated by Natalia Ratnikova over 4 years ago

[cmsstor152.fnal.gov] (local) admin > logoff
dmg.util.CommandExitException: (0) Done
Connection to cmsstor152.fnal.gov closed.
[root@cmsstor152 ~]# dcache stop
Stopping dCacheDomain 0 1 done
Stopping adminDoorDomain 0 done
Stopping httpdDomain 0 1 done
Stopping billingDomain 0 1 done
Stopping dirDomain 0 done

867  dcache stop
868 rpm -q java-1.7.0-openjdk
869 yum erase jdk
870 yum --disablerepo=osg --disablerepo=uscmst1 -y install java-1.7.0-openjdk
cp -p /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security.orig

[root@cmsstor152 ~]# perl p -i -e 's/^(jdk.certpath.disabledAlgorithms=MD2)(, RSA .*$)/$1/ && print "#" . $1 . $2 . "\n"' /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
[root@cmsstor152 ~]# diff /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security.orig
416,417c416
< #jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024
< jdk.certpath.disabledAlgorithms=MD2
--

jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024

[root@cmsstor152 ~]#

[root@cmsstor152 ~]# perl p -i -e 's/^(jdk.certpath.disabledAlgorithms=MD2)(, RSA .*$)/$1/ && print "#" . $1 . $2 . "\n"' /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security
[root@cmsstor152 ~]# diff /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.85.x86_64/jre/lib/security/java.security.orig
416,417c416
< #jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024
< jdk.certpath.disabledAlgorithms=MD2
--

jdk.certpath.disabledAlgorithms=MD2, RSA keySize < 1024

#11 Updated by Natalia Ratnikova over 4 years ago

[root@cmsstor152 ~]# java -version
java version "1.7.0_85"
OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)
OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)

[root@cmsstor152 ~]# dcache status
DOMAIN STATUS PID USER
dirDomain stopped root
billingDomain stopped root
httpdDomain stopped root
adminDoorDomain stopped root
dCacheDomain stopped root
[root@cmsstor152 ~]# java -version
java version "1.7.0_85"
OpenJDK Runtime Environment (rhel-2.6.1.3.el6_6-x86_64 u85-b01)
OpenJDK 64-Bit Server VM (build 24.85-b03, mixed mode)
[root@cmsstor152 ~]# dcache start
Starting dirDomain done
Starting billingDomain done
Starting httpdDomain done
Starting adminDoorDomain done
Starting dCacheDomain done
[root@cmsstor152 ~]# ssh -1 -i /etc/dcache/admin/server_key -c blowfish -p 22223 cmsstor152.fnal.gov -l admin

dCache Admin (VII) (user=admin)

[cmsstor152.fnal.gov] (local) admin > cd PoolManager
[cmsstor152.fnal.gov] (PoolManager) admin > psu ls pool
w-cmsstor155-disk_itb-disk1
w-cmsstor112-disk_itb-disk3
w-cmsstor112-disk_itb-disk1
w-cmsstor112-disk_itb-disk2
w-cmsstor151-disk_itb-disk2
w-cmsstor151-disk_itb-disk3
w-cmsstor155-disk_itb-disk2
w-cmsstor155-disk_itb-disk3
w-cmsstor151-disk_itb-disk1
[cmsstor152.fnal.gov] (PoolManager) admin > ..
[cmsstor152.fnal.gov] (local) admin > cd w-cmsstor112-disk_itb-disk1
[cmsstor152.fnal.gov] (w-cmsstor112-disk_itb-disk1) admin > migration ls
[1] RUNNING migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
[cmsstor152.fnal.gov] (w-cmsstor112-disk_itb-disk1) admin > migration info 1
Command : migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
State : RUNNING
Queued : 0
Attempts : 20
Targets : w-cmsstor112-disk_itb-disk3,w-cmsstor112-disk_itb-disk2
Completed : 3 files; 3221225472 bytes; 21%
Total : 15032385536 bytes
Concurrency: 10
Running tasks:
[16] 0000942412B618E74064B81A34F65D0367ED: TASK.GettingLocations
[17] 0000D4A7C6DAB3564DDC808619C485CC2DDB: TASK.GettingLocations
[18] 0000574C2EA17E1243249A086E1EA47BCC77: TASK.GettingLocations
[19] 0000FEC03B015C2F41D7A1E3E96C9017CADE: TASK.GettingLocations
Most recent errors:
18:21:16 [1] 00007E6F3D9CABC049958AE490936FFCB71E: Replica is locked on target pool
18:37:11 [5] 0000D4A7C6DAB3564DDC808619C485CC2DDB: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:37:11 [7] 0000942412B618E74064B81A34F65D0367ED: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:37:41 [4] 0000574C2EA17E1243249A086E1EA47BCC77: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:38:11 [6] 0000FEC03B015C2F41D7A1E3E96C9017CADE: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:42:41 [9] 0000942412B618E74064B81A34F65D0367ED: PnfsManager failed (no response)
18:42:41 [8] 0000D4A7C6DAB3564DDC808619C485CC2DDB: PnfsManager failed (no response)
18:43:11 [10] 0000574C2EA17E1243249A086E1EA47BCC77: PnfsManager failed (no response)
18:43:41 [11] 0000FEC03B015C2F41D7A1E3E96C9017CADE: PnfsManager failed (no response)
18:48:11 [12] 0000942412B618E74064B81A34F65D0367ED: PnfsManager failed (no response)
18:48:11 [13] 0000D4A7C6DAB3564DDC808619C485CC2DDB: PnfsManager failed (no response)
18:48:41 [14] 0000574C2EA17E1243249A086E1EA47BCC77: PnfsManager failed (no response)
18:49:11 [15] 0000FEC03B015C2F41D7A1E3E96C9017CADE: PnfsManager failed (no response)
[cmsstor152.fnal.gov] (w-cmsstor112-disk_itb-disk1) admin > migration info 1
Command : migration copy verify -concurrency=10 -tmode=cached -pins=keep -permanent -target=pgroup - burningPools
State : RUNNING
Queued : 0
Attempts : 20
Targets : w-cmsstor112-disk_itb-disk3,w-cmsstor112-disk_itb-disk2
Completed : 3 files; 3221225472 bytes; 21%
Total : 15032385536 bytes
Concurrency: 10
Running tasks:
[16] 0000942412B618E74064B81A34F65D0367ED: TASK.GettingLocations
[17] 0000D4A7C6DAB3564DDC808619C485CC2DDB: TASK.GettingLocations
[18] 0000574C2EA17E1243249A086E1EA47BCC77: TASK.GettingLocations
[19] 0000FEC03B015C2F41D7A1E3E96C9017CADE: TASK.GettingLocations
Most recent errors:
18:21:16 [1] 00007E6F3D9CABC049958AE490936FFCB71E: Replica is locked on target pool
18:37:11 [5] 0000D4A7C6DAB3564DDC808619C485CC2DDB: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:37:11 [7] 0000942412B618E74064B81A34F65D0367ED: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:37:41 [4] 0000574C2EA17E1243249A086E1EA47BCC77: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:38:11 [6] 0000FEC03B015C2F41D7A1E3E96C9017CADE: Pool [w-cmsstor112-disk_itb-disk2@local] failed (no response)
18:42:41 [9] 0000942412B618E74064B81A34F65D0367ED: PnfsManager failed (no response)
18:42:41 [8] 0000D4A7C6DAB3564DDC808619C485CC2DDB: PnfsManager failed (no response)
18:43:11 [10] 0000574C2EA17E1243249A086E1EA47BCC77: PnfsManager failed (no response)
18:43:41 [11] 0000FEC03B015C2F41D7A1E3E96C9017CADE: PnfsManager failed (no response)
18:48:11 [12] 0000942412B618E74064B81A34F65D0367ED: PnfsManager failed (no response)
18:48:11 [13] 0000D4A7C6DAB3564DDC808619C485CC2DDB: PnfsManager failed (no response)
18:48:41 [14] 0000574C2EA17E1243249A086E1EA47BCC77: PnfsManager failed (no response)
18:49:11 [15] 0000FEC03B015C2F41D7A1E3E96C9017CADE: PnfsManager failed (no response)
[cmsstor152.fnal.gov] (w-cmsstor112-disk_itb-disk1) admin >

#12 Updated by Natalia Ratnikova over 4 years ago

[root@cmsadmin1 ~]# for n in cmsstor112 cmsstor151 cmsstor152 cmsstor153 cmsstor155 cmspnfs1; do echo $n; ssh $n 'service postgresql-9.2 status'; done
cmsstor112
postgresql-9.2: unrecognized service
cmsstor151
postgresql-9.2: unrecognized service
cmsstor152
is stopped
cmsstor153
is stopped
cmsstor155
postgresql-9.2: unrecognized service
cmspnfs1
is stopped
[root@cmsadmin1 ~]# for n in cmsstor112 cmsstor151 cmsstor152 cmsstor153 cmsstor155 cmspnfs1; do echo $n; ssh $n 'service dcache-server status'; done
cmsstor112
DOMAIN STATUS PID USER
w-cmsstor112-disk_itb-disk1Domain stopped root
w-cmsstor112-disk_itb-disk2Domain stopped root
w-cmsstor112-disk_itb-disk3Domain stopped root
gridftp-cmsstor112Domain stopped root
cmsstor151
DOMAIN STATUS PID USER
w-cmsstor151-disk_itb-disk1Domain stopped root
w-cmsstor151-disk_itb-disk2Domain stopped root
w-cmsstor151-disk_itb-disk3Domain stopped root
gridftp-cmsstor151Domain stopped root
cmsstor152
DOMAIN STATUS PID USER
dirDomain stopped root
billingDomain stopped root
httpdDomain stopped root
adminDoorDomain stopped root
dCacheDomain stopped root
cmsstor153
DOMAIN STATUS PID USER
nfsDomain.v3 stopped root
dcap-cmsstor153Domain stopped root
authdcap-cmsstor153Domain stopped root
gsidcap-cmsstor153Domain stopped root
xrootd-cmsstor153Domain stopped root
xrootdLFNs-cmsstor153Domain stopped root
utilityDomain stopped root
gPlazmaDomain stopped root
srm-cmsstor153Domain stopped root
transfermanagersDomain stopped root
cmsstor155
DOMAIN STATUS PID USER
w-cmsstor155-disk_itb-disk1Domain stopped root
w-cmsstor155-disk_itb-disk2Domain stopped root
w-cmsstor155-disk_itb-disk3Domain stopped root
gridftp-cmsstor155Domain stopped root
cmspnfs1
DOMAIN STATUS PID USER
namespaceDomain stopped root
nfsDomain stopped root
[root@cmsadmin1 ~]#
[root@cmsadmin1 ~]# for n in cmsstor112 cmsstor151 cmsstor152 cmsstor153 cmsstor155 cmspnfs1; do echo $n; ssh $n 'service puppet status'; done
cmsstor112
puppet is stopped
cmsstor151
puppet is stopped
cmsstor152
puppet is stopped
cmsstor153
puppet is stopped
cmsstor155
puppet is stopped
cmspnfs1
puppet is stopped
[root@cmsadmin1 ~]#

#13 Updated by Natalia Ratnikova over 4 years ago

  • Status changed from Accepted to Resolved

For the record:
command to check network driver and firmware version installed on the node:

ethtool -i eth0   # some dcache pools have eth1, 2,3, or even 4 (?) configured

Example:
[root@cmsstor327 ~]# ethtool -i eth0
driver: qlcnic
version: 5.3.59
firmware-version: 4.12.10
bus-info: 0000:04:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: no



Also available in: Atom PDF