DCSO » EOS

Obtaining certs

If needed, certs can be obtained via
User - https://oim-itb.grid.iu.edu/oim/certificaterequestuser
Host/Service - https://oim-itb.grid.iu.edu/oim/certificaterequesthost

Please let me know if you have any related questions/concerns.

I've requested 2 certs:

[root@cmsstor150 ~]# umask 077; openssl req -new -newkey rsa:2048 -nodes -keyout hostkey.pem -subj "/CN=`hostname --fqdn`"

and

[root@cmssrv151 grid-security]# umask 077; openssl req -new -newkey rsa:2048 -nodes -keyout cmseoskey-cilogon.pem -subj "/CN=cmssrv152.fnal.gov"

Certificates downloaded and added in the secret repo certs:

mac-124183:Downloads gerard1$ scp host_certificate.H6486.cmsstor150.fnal.gov.x509.pem root@cmsadmin1.fnal.gov:/srv/secrets/cmsstor150.fnal.gov/hostcert.pem
host_certificate.H6486.cmsstor150.fnal.gov.x509.pem 100% 1532 1.5KB/s 00:00
mac-124183:Downloads gerard1$ scp host_certificate.H6487.cmssrv152.fnal.gov.x509.pem root@cmsadmin1.fnal.gov:/srv/secrets/cmssrv152.fnal.gov/hostcert.pem
host_certificate.H6487.cmssrv152.fnal.gov.x509.pem 100% 1528 1.5KB/s 00:00
mac-124183:Downloads gerard1$

And moving the keys there too:

[root@cmsstor150 ~]# scp hostkey.pem root@cmsadmin1.fnal.gov:/srv/secrets/cmsstor150.fnal.gov/
hostkey.pem 100% 1704 1.7KB/s 00:00
[root@cmsstor150 ~]#
[root@cmssrv151 grid-security]# scp cmseoskey-cmssrv152-cilogon.pem root@cmsadmin1.fnal.gov:/srv/secrets/cmssrv152.fnal.gov/hostkey.pem
cmseoskey-cmssrv152-cilogon.pem 100% 1704 1.7KB/s 00:00
[root@cmssrv151 grid-security]#

Lisa pushed some of my changes at the same time while I was doing this so to see the actual changes I'm making something a bit funny:

[root@cmsadmin1 secrets]# git diff --name-status f51a18aa179f967ac6af30103343be50bddb72a6 | grep -v cmseos
M cmssrv152.fnal.gov/hostcert.pem
A cmssrv152.fnal.gov/hostcert.pem.old
M cmssrv152.fnal.gov/hostkey.pem
A cmssrv152.fnal.gov/hostkey.pem.old
M cmsstor150.fnal.gov/hostcert.pem
A cmsstor150.fnal.gov/hostcert.pem.old
M cmsstor150.fnal.gov/hostkey.pem
A cmsstor150.fnal.gov/hostkey.pem.old
[root@cmsadmin1 secrets]#

Now I'll use puppet to roll this out. Note that I don't fully remember the status of pupept deployment for this certs so I may have to do something additional...

First trying to get the certs pushed out to cmssrv151:

[root@cmssrv151 grid-security]# mkdir oldcerts
[root@cmssrv151 grid-security]# ll *.pem*
-rw-r--r-- 1 root root 2022 Apr 27  2015 cmseoscert.pem
-rw------- 1 root root 1704 Nov 30 13:44 cmseoskey-cmssrv152-cilogon.pem
-r-------- 1 root root 1675 Apr 27  2015 cmseoskey.pem
-rw-r--r-- 1 root root 2022 Jan 20  2015 hostcert.pem
-rw-r--r-- 1 root root 2022 Dec 10  2013 hostcert.pem.OLD
-r-------- 1 root root 1675 Jan 20  2015 hostkey.pem
-r-------- 1 root root 1679 Dec 10  2013 hostkey.pem.OLD
[root@cmssrv151 grid-security]# mv *.pem* oldcerts/
[root@cmssrv151 grid-security]# ll
total 408
drwxr-xr-x 2 daemon daemon   4096 Jun 24 12:17 bestman
drwxr-xr-x 2 daemon daemon   4096 Apr 27  2015 bestman2
lrwxrwxrwx 1 root   root       20 Nov  3 18:44 certificates -> certificates-1.50NEW
drwxr-xr-x 2 root   root    53248 Oct 13 14:24 certificates-1.49NEW.old
drwxr-xr-x 2 root   root    53248 Nov  3 18:48 certificates-1.50NEW
drwxr-xr-x 2 daemon daemon   4096 Jun 24 12:15 daemon
-rw-r--r-- 1 root   root   278535 Dec  8 13:15 grid-mapfile
drwxr-xr-x 2 root   root     4096 Dec  8 16:39 oldcerts
drwxr-xr-x 2 root   root     4096 Apr 28  2015 vomsdir
[root@cmssrv151 grid-security]# ll bestman*
bestman:
total 8
-rw-r--r-- 1 bestman bestman 2022 Jun 24 12:17 bestmancert.pem
-r-------- 1 bestman bestman 1679 Jun 24 12:15 bestmankey.pem

bestman2:
total 0
lrwxrwxrwx 1 daemon daemon 42 Apr 27  2015 hostcert.pem -> /etc/grid-security/bestman/bestmancert.pem
lrwxrwxrwx 1 daemon daemon 41 Apr 27  2015 hostkey.pem -> /etc/grid-security/bestman/bestmankey.pem
[root@cmssrv151 grid-security]# mv bestman oldcerts/
[root@cmssrv151 grid-security]# puppet agent -t
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts in /var/lib/puppet/lib/facter/afs_cache_size.rb
Info: Loading facts in /var/lib/puppet/lib/facter/os_maj_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/cvmfspartsize.rb
Info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/pe_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/concat_basedir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/condorceversion.rb
Info: Loading facts in /var/lib/puppet/lib/facter/postgres_default_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/rsyslog_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/code_server.rb
Info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
Info: Loading facts in /var/lib/puppet/lib/facter/iptables_persistent_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/certificate_facts.rb
Info: Loading facts in /var/lib/puppet/lib/facter/puppet_vardir.rb
Info: Loading facts in /var/lib/puppet/lib/facter/ip6tables_version.rb
Info: Loading facts in /var/lib/puppet/lib/facter/cvmfsversion.rb
Info: Caching catalog for cmssrv151.fnal.gov
Info: Applying configuration version '1449614349'
Notice: /Stage[main]/P_grid_certificate/P_secret::File[/etc/grid-security/hostkey.pem]/File[/etc/grid-security/hostkey.pem]/ensure: defined content as '{md5}0369b0729d286401e87d65692f9a0e39'
Notice: /Stage[main]/P_grid_certificate/P_secret::File[/etc/grid-security/hostcert.pem]/File[/etc/grid-security/hostcert.pem]/ensure: defined content as '{md5}522248c471a406067757922a0c9b24ea'
Notice: /Stage[main]/Eos::Gridcerts/File[/etc/grid-security/bestman]/ensure: created
Notice: /Stage[main]/Eos::Gridcerts/File[/etc/grid-security/bestman/bestmancert.pem]/ensure: defined content as '{md5}b846148ae1ca7ab7b0ab2129e8795526'
Notice: /Stage[main]/Eos::Gridcerts/File[/etc/grid-security/bestman/bestmankey.pem]/ensure: defined content as '{md5}43e3a9b564ef528c7109233254abca00'
Notice: Finished catalog run in 25.78 seconds
[root@cmssrv151 grid-security]# 

Checking the deployed certs:

[root@cmssrv151 grid-security]# openssl x509 -noout -text -in /etc/grid-security/hostcert.pem | grep Subject:
        Subject: DC=com, DC=DigiCert-Grid, O=Open Science Grid, OU=Services, CN=cmssrv151.fnal.gov
[root@cmssrv151 grid-security]# 
[root@cmssrv151 grid-security]# openssl x509 -noout -text -in /etc/grid-security/bestman/bestmancert.pem | grep Subject:
        Subject: DC=com, DC=DigiCert-Grid, O=Open Science Grid, OU=Services, CN=cmssrv152.fnal.gov
[root@cmssrv151 grid-security]# 

So.. Failure, the proper certs have not been pushed!

Ups... I had to do this on cmssrv153! Going for that.

Checking on the (simpler) FST node cmsstor150.

The cert here was deployed a week ago, when I pushed it to the secrets repo:

[root@cmsstor150 ~]# ls -ltrah /etc/grid-security/
total 412K
drwxr-xr-x    2 root root  52K Oct 14 08:47 certificates-1.49NEW.old
lrwxrwxrwx    1 root root   20 Nov  3 17:14 certificates -> certificates-1.50NEW
drwxr-xr-x    2 root root  52K Nov  3 18:43 certificates-1.50NEW
-r--------    1 root root 1.7K Dec  1 17:02 hostkey.pem
-rw-r--r--    1 root root 1.5K Dec  1 17:02 hostcert.pem
drwxr-xr-x. 110 root root  12K Dec  3 03:29 ..
-rw-r--r--    1 root root 273K Dec  8 13:15 grid-mapfile
drwxr-xr-x.   4 root root 4.0K Dec  8 13:15 .
[root@cmsstor150 ~]# openssl x509 -noout -text -in /etc/grid-security/hostcert.pem | grep Subject:
        Subject: DC=org, DC=opensciencegrid, O=Open Science Grid, OU=Services, CN=cmsstor150.fnal.gov
[root@cmsstor150 ~]# openssl x509 -noout -text -in /etc/grid-security/hostcert.pem | grep Issuer:
        Issuer: DC=org, DC=cilogon, C=US, O=CILogon, CN=CILogon OSG CA 1

On the other headnode the change didn't work very well either:

[root@cmssrv153 grid-security]# openssl x509 -noout -text -in /etc/grid-security/bestman/bestmancert.pem | grep Subject:
        Subject: DC=com, DC=DigiCert-Grid, O=Open Science Grid, OU=Services, CN=cmssrv152.fnal.gov
[root@cmssrv153 grid-security]# openssl x509 -noout -text -in /etc/grid-security/hostcert.pem | grep Subject:
        Subject: DC=com, DC=DigiCert-Grid, O=Open Science Grid, OU=Services, CN=cmssrv153.fnal.gov
[root@cmssrv153 grid-security]# 

I will test to check if the current gridftp transfer works at least, going to an (old digicert) bestman but to a CILogon gridftp server.... It's a good test because apparently this never worked! The poor test bestman is setup thinking it's the production one, see error:

[srminfo]Unrecognizable url:srm://cmssrv152.fnal.gov:8443/srm/v2/server?SFN=/eos/cmseos-test.fnal.gov/data/gerard/bin.tar for this server:httpg://cmseos.fnal.gov:8443/srm/v2/server

This is on cmssrv153 when trying the following from cmsphedex-disk as cmsprod:

-bash-4.1$ export X509_USER_PROXY=/home/cmsprod/phedex/gridcert/proxy.cert
-bash-4.1$ srmcp -debug -2 srm://cmssrv152.fnal.gov:8443/srm/v2/server?SFN=/eos/cmseos-test.fnal.gov/data/gerard/bin.tar file:///tmp/test.gba

After a decent amount of bug-fixing and hackery now this works:

-bash-4.1$ srmcp -debug -2 srm://cmssrv152.fnal.gov:8443/srm/v2/server?SFN=/eos/test/gba.test.touch file:////tmp/test.gba.$$

-bash-4.1$ globus-url-copy -dbg  gsiftp://cmsstor150.fnal.gov//eos/test/gba.test.touch file:////home/cmsprod/myfile.test.gba.gftp

All the hackery is done manual, with puppet disabled, on cmssrv153 (EOS MGM and bestman server), need to puppetize it all:

1. Add cmsstor150.fnal.gov in the list of 'unix' authorized hosts in /etc/xrd.cf.mgm
2. change /etc/bestman2/conf/bestman2.rc so that

[root@cmssrv153 grid-security]# grep cmssrv152 /etc/sysconfig/bestman2 /etc/bestman2/conf/bestman2.rc 
/etc/sysconfig/bestman2:GLOBUS_HOSTNAME=cmssrv152.fnal.gov
/etc/bestman2/conf/bestman2.rc:supportedProtocolList=gsiftp://cmsstor150.fnal.gov;srm://cmseos-test.fnal.gov;srm://cmssrv152.fnal.gov;srm://cmssrv153.fnal.gov;srm://cmssrv151.fnal.gov

#AND

localPathListAllowed=/eos/uscms;/lustre/unmerged;/eos/test

3. Actually I should stop using cmssrv152 and get cmseos-test.fnal.gov on (need to get cert, the ip is already there!).