Project

General

Profile

Bug #9234

Updating EOS mount sensor

Added by Gerard Bernabeu Altayo over 4 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Start date:
06/17/2015
Due date:
06/17/2015
% Done:

100%

Estimated time:
1.00 h
Duration: 1

Description

Need to improve the sensor because the restart was leaving the eosd daemon dead.

History

#1 Updated by Gerard Bernabeu Altayo over 4 years ago

Thanks Natalia,

I could not find anything in the foswiki about this (https://cmsweb.fnal.gov/bin/view/Monitoring/WebSearch?tab=search&search=uscms_zabbix_plugins&scope=all&web=all&recurse=on), but as you said the history can be resonstructed somehow:

 809  rm -rf uscms_zabbix_plugins-1.8.17
  810  cd /usr/src/redhat/
  811  ls
  812  cd SOURCES
  813  ls
  814  cp uscms_zabbix_plugins-1.8.18.tar.gz /root/ma
  815  cd /root/ma
 832  tar xvzf uscms_zabbix_plugins-1.8.18.tar.gz
  833  ls -ltr
  834  cd uscms_zabbix_plugins-1.8.18
  835  ls
  836  cd plugins/
  837  ls
  838  cd src
  839  ls
  840  ls -ltr
  841  scp root@cmsadmin1:/root/check_eos* /tmp
  842  diff check_eos_mount /tmp/check_eos_mount
  843  ls
  844  cp /tmp/check_eos_mount .
  845  ls
  867  rm uscms_zabbix_plugins-1.8.18.tar.gz
  868  history |grep tar
  869  tar cvzf uscms_zabbix_plugins-1.8.18.tar.gz uscms_zabbix_plugins-1.8.18/
  870  ls -ltr
  871  cp uscms_zabbix_plugins-1.8.18.tar.gz /usr/src/redhat/SOURCES/
  872  cd /usr/src/redhat/
  873  ls
  874  cd SPECS
  878  history |grep rpm
  879  rpmbuild -ba --target=noarch uscms_zabbix_plugins.spec
  880  cd /root/rpmbuild/
  881  ls
  882  cd SOURCES
  883  ls
  884  pwd
  885  cd -
  886  cd /usr/src/
#A few (failed?) attempts to build the rpm
 912  cp ../SOURCES/uscms_zabbix_plugins-1.8.18.tar.gz /root/rpmbuild/SOURCES
  913  rpmbuild -ba --target=noarch uscms_zabbix_plugins.spec
  914  cd /root/rpmbuild/
  939  cd /root/rpmbuild/
  940  ls
  941  cd RP
  942  cd RPMS/noarch/
  943  ls
  944  scp uscms_zabbix_plugins-1.8.18-2.noarch.rpm root@cmsadmin1:/root
  947  rpm -Uvh /tmp/uscms_zabbix_plugins-1.8.18-2.noarch.rpm

Will give it a try.... This seems to work:

[root@cmszabbix1 ma]# vim /usr/src/redhat/SPECS/uscms_zabbix_plugins.spec #Increase version number to 1.8.19

[root@cmszabbix1 ~]# ll /usr/src/redhat/SOURCES/uscms_zabbix_plugins-1.8.18.tar.gz
-rw-r--r-- 1 root root 47035 May 28 13:19 /usr/src/redhat/SOURCES/uscms_zabbix_plugins-1.8.18.tar.gz
[root@cmszabbix1 ~]# cd /usr/src/redhat/SOURCES/
[root@cmszabbix1 SOURCES]# tar -xvf /usr/src/redhat/SOURCES/uscms_zabbix_plugins-1.8.18.tar.gz
[root@cmszabbix1 SOURCES]# mv uscms_zabbix_plugins-1.8.18 uscms_zabbix_plugins-1.8.19

[root@cmslpc33 ~]# scp /usr/local/bin/check_eos_mount cmszabbix1.fnal.gov:/usr/src/redhat/SOURCES/uscms_zabbix_plugins-1.8.19/plugins/src/check_eos_mount
check_eos_mount                                                                                                                                    100% 5007     4.9KB/s   00:00   
[root@cmslpc33 ~]#

[root@cmszabbix1 SOURCES]# tar -czf uscms_zabbix_plugins-1.8.19.tar.gz uscms_zabbix_plugins-1.8.19
[root@cmszabbix1 SPECS]# cd /usr/src/redhat/SPECS
[root@cmszabbix1 SPECS]# cp /usr/src/redhat/SOURCES/uscms_zabbix_plugins-1.8.19.tar.gz /root/rpmbuild/SOURCES/uscms_zabbix_plugins-1.8.19.tar.gz  #Not sure why I have to do this...
[root@cmszabbix1 SPECS]# rpmbuild -ba --target=noarch uscms_zabbix_plugins.spec
[root@cmszabbix1 SPECS]# ll /root/rpmbuild/RPMS/noarch/uscms_zabbix_plugins-1.8.19-1.noarch.rpm
-rw-r--r-- 1 root root 52270 Jun 17 15:16 /root/rpmbuild/RPMS/noarch/uscms_zabbix_plugins-1.8.19-1.noarch.rpm

[root@cmszabbix1 SPECS]# rpm -vUh /root/rpmbuild/RPMS/noarch/uscms_zabbix_plugins-1.8.19-1.noarch.rpm
Preparing...                ########################################### [100%]
   1:uscms_zabbix_plugins   ########################################### [100%]
[root@cmszabbix1 SPECS]# rpm -q uscms_zabbix_plugins
uscms_zabbix_plugins-1.8.19-1.noarch
[root@cmszabbix1 SPECS]# vim /usr/local/bin/check_eos_mount
[root@cmszabbix1 SPECS]# grep -i version /usr/local/bin/check_eos_mount
# Version: 3.4

So this works!

Will now put it in the repo:

ssh root@cms-install.fnal.gov
scp root@cmszabbix1.fnal.gov:/root/rpmbuild/RPMS/noarch/uscms_zabbix_plugins-1.8.19-1.noarch.rpm /srv/repo/slf6-x86_64/
cd /srv/repo
make uscmst1

Testing it now:

[root@cmslpc37 ~]# yum clean all --disablerepo=* --enablerepo=uscmst1
Loaded plugins: priorities, security
Cleaning repos: uscmst1
Cleaning up Everything
[root@cmslpc37 ~]# yum update --exclude=kernel*
Loaded plugins: priorities, security
Setting up Update Process
uscmst1                                                                                                                                                       | 2.5 kB     00:00    
uscmst1/primary_db                                                                                                                                            | 138 kB     00:00    
7749 packages excluded due to repository priority protections
Resolving Dependencies
--> Running transaction check
---> Package uscms_zabbix_plugins.noarch 0:1.8.18-2 will be updated
---> Package uscms_zabbix_plugins.noarch 0:1.8.19-1 will be an update
--> Finished Dependency Resolution

Adding all this in https://cdcvs.fnal.gov/redmine/issues/9234

For the SL5 machines I don't see this RPM, not sure how the check should be distributed....

Gerard

#2 Updated by Gerard Bernabeu Altayo over 4 years ago

  • Due date set to 06/17/2015
  • Status changed from New to Resolved
  • % Done changed from 0 to 100
  • Estimated time set to 1.00 h

Update done on the SL5 nodes manually:

[root@cmslpc33 ~]# scp cmslpc37:/usr/local/bin/check_eos_mount /usr/local/bin/check_eos_mount
check_eos_mount 100% 5007 4.9KB/s 00:00
[root@cmslpc33 ~]# for i in `seq 28 35`; do rsh cmslpc$i uname -a; done
Linux cmslpc28.fnal.gov 2.6.18-400.1.1.el5 #1 SMP Wed Dec 17 14:22:42 CST 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux cmslpc29.fnal.gov 2.6.18-400.1.1.el5 #1 SMP Wed Dec 17 14:22:42 CST 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux cmslpc30.fnal.gov 2.6.18-400.1.1.el5 #1 SMP Wed Dec 17 14:22:42 CST 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux cmslpc31.fnal.gov 2.6.18-400.1.1.el5 #1 SMP Wed Dec 17 14:22:42 CST 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux cmslpc32.fnal.gov 2.6.18-400.1.1.el5 #1 SMP Wed Dec 17 14:22:42 CST 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux cmslpc33.fnal.gov 2.6.18-400.1.1.el5 #1 SMP Wed Dec 17 14:22:42 CST 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux cmslpc34.fnal.gov 2.6.18-400.1.1.el5 #1 SMP Wed Dec 17 14:22:42 CST 2014 x86_64 x86_64 x86_64 GNU/Linux
Linux cmslpc35.fnal.gov 2.6.18-398.el5 #1 SMP Tue Sep 16 01:03:15 CDT 2014 x86_64 x86_64 x86_64 GNU/Linux

[root@cmslpc33 ~]# for i in `seq 28 35`; do rsh -F cmslpc$i scp cmslpc37:/usr/local/bin/check_eos_mount /usr/local/bin/check_eos_mount; done
[root@cmslpc33 ~]#

The SL6 machines will pickup the new version from the new RPM :).

#3 Updated by Gerard Bernabeu Altayo over 4 years ago

I actually didn't make the script in a way that zabbix like the way we have it deployed, had to update and redeploy with:

-bash-4.1$ pssh -l root --verbose --outdir /tmp/pssh_output.gba --timeout 600 -h /usr/local/etc/farmlets/all 'yum clean all --disablerepo=* --enablerepo=uscmst1; yum update -y uscms_zabbix_plugins'

And then "rerun" zabbix like:

-bash-4.1$ pssh -l root --verbose --outdir /tmp/pssh_output.gba --timeout 600 -h /usr/local/etc/farmlets/workers 'service zabbix-agent restart'



Also available in: Atom PDF