Project

General

Profile

Bug #11808

glibc EOS upgrade

Added by Gerard Bernabeu Altayo over 3 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Start date:
02/24/2016
Due date:
% Done:

0%

Estimated time:
Duration:

Description

We need to upgrade all EOS nodes for glibc, reboot included. We have a whole day downtime so we can be disruptive.

History

#1 Updated by Gerard Bernabeu Altayo over 3 years ago

I did the compactification on cmssrv222 and then upgrade& reboot.

Doing upgrade&reboot to the standby cmssrv238 and all the FSTs:

rgang_ssh eosfst.2016024 'uptime'
rgang_ssh eosfst.2016024 'yum clean all; yum update -y --exclude=xrootd*'

[root@cmsadmin1 gerard1]# rgang_ssh eosfst.2016024 'rpm -q glibc-2.12-1.166.el6_7.7.x86_64'
cmseos12.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos11.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos13.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos14.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos15.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos16.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos17.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos1.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos5.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos4.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos3.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos2.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos25.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos26.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos27.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos28.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos29.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos30.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos31.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos32.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos33.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos39.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos34.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos36.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos34.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos38.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos37.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos35.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos42.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos40.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos41.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos43.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos44.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos45.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos46.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos47.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos49.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos48.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos50.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos51.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos52.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos53.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos56.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos54.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos55.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos57.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos58.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos59.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos60.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos61.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos62.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64
cmseos63.fnal.gov= glibc-2.12-1.166.el6_7.7.x86_64

#2 Updated by Gerard Bernabeu Altayo over 3 years ago

As it's getting too usual, the MGM (head node) does NOT want to start fine... This is an old known issue, it is trying to start as slave and it fails:

160224 09:47:26 time=1456328846.110097 func=BootNamespace            level=ALERT logid=e5a3ae8e-db0d-11e5-9468-a0369f23d048 unit=mgm@cmssrv222.fnal.gov:1094 tid=00007f375b3c4740 source=Master:1861                    tident=<service> sec=      uid=0 gid=0 name= geo="" msg="running boot sequence (as slave)" 
160224 09:47:26 time=1456328846.146342 func=BootNamespace            level=NOTE  logid=e5a3ae8e-db0d-11e5-9468-a0369f23d048 unit=mgm@cmssrv222.fnal.gov:1094 tid=00007f375b3c4740 source=Master:1920                    tident=<service> sec=      uid=0 gid=0 name= geo="" eos directory view configure started as slave
160224 09:47:26 time=1456328846.146639 func=BootNamespace            level=CRIT  logid=e5a3ae8e-db0d-11e5-9468-a0369f23d048 unit=mgm@cmssrv222.fnal.gov:1094 tid=00007f375b3c4740 source=Master:1975                    tident=<service> sec=      uid=0 gid=0 name= geo="" eos view initialization failed after 0 seconds
160224 09:47:26 time=1456328846.146669 func=BootNamespace            level=CRIT  logid=e5a3ae8e-db0d-11e5-9468-a0369f23d048 unit=mgm@cmssrv222.fnal.gov:1094 tid=00007f375b3c4740 source=Master:1977                    tident=<service> sec=      uid=0 gid=0 name= geo="" initialization returned ec=3 Unable to read the magic number from: /var/eos/md/directories.cmssrv238.fnal.gov.mdlog
160224 09:47:26 10292 XrootdConfig: Unable to create file system object via libXrdEosMgm.so
160224 09:47:26 10292 XrootdConfig: Unable to load file system.
------ xrootd protocol initialization failed.
160224 09:47:26 10292 XrdProtocol: Protocol xrootd could not be loaded
------ xrootd mgm@cmssrv222.fnal.gov:-1 initialization failed.

Following the instructions in http://eos.readthedocs.org/en/latest/configuration/master.html

[root@cmssrv222 ~]# service eos stop
Stopping xrootd: mgm                                       [  OK  ]
Stopping xrootd: mq                                        [  OK  ]
Stopping xrootd: sync                                      [  OK  ]
[root@cmssrv222 ~]# service eos master mgm
Configured MGM on localhost as master                      [  OK  ]
[root@cmssrv222 ~]# 
[root@cmssrv222 ~]# service eos master mq
Configured MQ on localhost as master                       [  OK  ]
[root@cmssrv222 ~]# service eos start

Starting xrootd as mgm with -n mgm -c /etc/xrd.cf.mgm -m -l /var/log/eos/xrdlog.mgm -b -Rdaemon

Now I am rebooting all the FSTs:

[root@cmsadmin1 gerard1]# rgang_ssh eosfst.2016024 'reboot'
cmseos12.fnal.gov= cmseos11.fnal.gov= cmseos13.fnal.gov= cmseos14.fnal.gov= cmseos15.fnal.gov= cmseos16.fnal.gov= cmseos17.fnal.gov= cmseos1.fnal.gov= cmseos5.fnal.gov= cmseos4.fnal.gov= cmseos3.fnal.gov= cmseos2.fnal.gov= cmseos25.fnal.gov= cmseos26.fnal.gov= cmseos27.fnal.gov= cmseos28.fnal.gov= cmseos29.fnal.gov= cmseos30.fnal.gov= cmseos31.fnal.gov= cmseos32.fnal.gov= cmseos33.fnal.gov= cmseos39.fnal.gov= cmseos34.fnal.gov= cmseos36.fnal.gov= cmseos34.fnal.gov= cmseos38.fnal.gov= cmseos37.fnal.gov= cmseos35.fnal.gov= cmseos42.fnal.gov= cmseos40.fnal.gov= cmseos41.fnal.gov= cmseos43.fnal.gov= cmseos44.fnal.gov= cmseos45.fnal.gov= cmseos46.fnal.gov= cmseos47.fnal.gov= cmseos49.fnal.gov= cmseos48.fnal.gov= cmseos50.fnal.gov= cmseos51.fnal.gov= cmseos52.fnal.gov= cmseos53.fnal.gov= cmseos56.fnal.gov= cmseos54.fnal.gov= cmseos55.fnal.gov= cmseos57.fnal.gov= cmseos58.fnal.gov= cmseos59.fnal.gov= cmseos60.fnal.gov= cmseos61.fnal.gov= cmseos62.fnal.gov= cmseos63.fnal.gov= [root@cmsadmin1 gerard1]# 

#3 Updated by Gerard Bernabeu Altayo over 3 years ago

A few FSTs are not booting right, others are still booting. The only ones that came online shortly after reboot are the empty ones (as expected).

Will be looking at the failed ones....

[root@cmssrv222 ~]# eos fs ls

#..........................................................................................................................................
#                   host (#...) #   id #           path #     schedgroup #         geotag #       boot # configstatus #      drain # active
#..........................................................................................................................................
       cmseos12.fnal.gov (1095)     41   /storage/data3        default.2                       booting             rw      nodrain   online
       cmseos12.fnal.gov (1095)     42   /storage/data1        default.3                       booting             rw      nodrain   online
       cmseos12.fnal.gov (1095)     43   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos11.fnal.gov (1095)     44   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos11.fnal.gov (1095)     45   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos11.fnal.gov (1095)     46   /storage/data3        default.2                       booting             rw      nodrain   online
       cmseos13.fnal.gov (1095)     47   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos13.fnal.gov (1095)     48   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos13.fnal.gov (1095)     49   /storage/data3        default.2                       booting             rw      nodrain   online
       cmseos14.fnal.gov (1095)     50   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos14.fnal.gov (1095)     51   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos14.fnal.gov (1095)     52   /storage/data3        default.3                   bootfailure             rw      nodrain   online
       cmseos15.fnal.gov (1095)     53   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos15.fnal.gov (1095)     54   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos15.fnal.gov (1095)     55   /storage/data3        default.3                       booting             rw      nodrain   online
       cmseos16.fnal.gov (1095)     56   /storage/data1        default.3                       booting             rw      nodrain   online
       cmseos16.fnal.gov (1095)     57   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos16.fnal.gov (1095)     58   /storage/data3        default.2                       booting             rw      nodrain   online
       cmseos17.fnal.gov (1095)     59   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos17.fnal.gov (1095)     60   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos17.fnal.gov (1095)     61   /storage/data3        default.2                       booting             rw      nodrain   online
        cmseos1.fnal.gov (1095)     62   /storage/data1        default.0                       booting             rw      nodrain   online
        cmseos1.fnal.gov (1095)     63   /storage/data2        default.1                       booting             rw      nodrain   online
        cmseos1.fnal.gov (1095)     64   /storage/data3        default.2                   bootfailure             rw      nodrain   online
        cmseos5.fnal.gov (1095)     65   /storage/data1        default.0                        booted             ro      nodrain   online
        cmseos5.fnal.gov (1095)     66   /storage/data2        default.1                        booted             ro      nodrain   online
        cmseos5.fnal.gov (1095)     67   /storage/data3        default.2                        booted             ro      nodrain   online
        cmseos4.fnal.gov (1095)     68   /storage/data1        default.0                       booting             ro      nodrain   online
        cmseos4.fnal.gov (1095)     69   /storage/data2        default.1                       booting             ro      nodrain   online
        cmseos4.fnal.gov (1095)     70   /storage/data3        default.2                       booting             ro      nodrain   online
        cmseos3.fnal.gov (1095)     71   /storage/data1        default.0                       booting             ro      nodrain   online
        cmseos3.fnal.gov (1095)     72   /storage/data2        default.2                       booting             ro      nodrain   online
        cmseos3.fnal.gov (1095)     73   /storage/data3        default.1                       booting             ro      nodrain   online
        cmseos2.fnal.gov (1095)     74   /storage/data1        default.0                       booting             ro      nodrain   online
        cmseos2.fnal.gov (1095)     75   /storage/data2        default.1                       booting             ro      nodrain   online
        cmseos2.fnal.gov (1095)     76   /storage/data3        default.3                        booted             ro      nodrain   online
       cmseos25.fnal.gov (1095)    103   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos25.fnal.gov (1095)    104   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos26.fnal.gov (1095)    105   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos26.fnal.gov (1095)    106   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos27.fnal.gov (1095)    107   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos27.fnal.gov (1095)    108   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos27.fnal.gov (1095)    109   /storage/data3        default.2                       booting             rw      nodrain   online
       cmseos28.fnal.gov (1095)    110   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos28.fnal.gov (1095)    111   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos29.fnal.gov (1095)    112   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos29.fnal.gov (1095)    113   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos30.fnal.gov (1095)    116   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos30.fnal.gov (1095)    117   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos31.fnal.gov (1095)    119   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos31.fnal.gov (1095)    120   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos32.fnal.gov (1095)    122   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos32.fnal.gov (1095)    123   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos33.fnal.gov (1095)    125   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos33.fnal.gov (1095)    126   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos39.fnal.gov (1095)    134   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos39.fnal.gov (1095)    135   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos34.fnal.gov (1095)    136   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos36.fnal.gov (1095)    137   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos36.fnal.gov (1095)    138   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos34.fnal.gov (1095)    139   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos38.fnal.gov (1095)    140   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos38.fnal.gov (1095)    141   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos37.fnal.gov (1095)    142   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos37.fnal.gov (1095)    143   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos35.fnal.gov (1095)    144   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos35.fnal.gov (1095)    145   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos42.fnal.gov (1095)    146   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos42.fnal.gov (1095)    147   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos40.fnal.gov (1095)    148   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos40.fnal.gov (1095)    149   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos41.fnal.gov (1095)    150   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos41.fnal.gov (1095)    151   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos43.fnal.gov (1095)    152   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos43.fnal.gov (1095)    153   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos44.fnal.gov (1095)    154   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos44.fnal.gov (1095)    155   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos45.fnal.gov (1095)    156   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos45.fnal.gov (1095)    157   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos46.fnal.gov (1095)    158   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos46.fnal.gov (1095)    159   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos47.fnal.gov (1095)    160   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos47.fnal.gov (1095)    161   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos49.fnal.gov (1095)    162   /storage/data1        default.2                       booting             rw      nodrain   online
       cmseos49.fnal.gov (1095)    163   /storage/data2        default.3                       booting             rw      nodrain   online
       cmseos48.fnal.gov (1095)    164   /storage/data1        default.0                       booting             rw      nodrain   online
       cmseos48.fnal.gov (1095)    165   /storage/data2        default.1                       booting             rw      nodrain   online
       cmseos50.fnal.gov (1095)    166   /storage/data1        default.0                        booted             ro      nodrain   online
       cmseos50.fnal.gov (1095)    167   /storage/data2        default.1                        booted             ro      nodrain   online
       cmseos51.fnal.gov (1095)    168   /storage/data1        default.2                        booted             ro      nodrain   online
       cmseos51.fnal.gov (1095)    169   /storage/data2        default.3                        booted             ro      nodrain   online
       cmseos52.fnal.gov (1095)    170   /storage/data1        default.0                        booted          drain     draining   online
       cmseos52.fnal.gov (1095)    171   /storage/data2        default.1                        booted          drain     draining   online
       cmseos53.fnal.gov (1095)    172   /storage/data1        default.2                        booted          drain     draining   online
       cmseos53.fnal.gov (1095)    173   /storage/data2        default.3                       booting          drain     draining   online
       cmseos56.fnal.gov (1095)    174   /storage/data1        default.0                       booting             ro      nodrain   online
       cmseos56.fnal.gov (1095)    175   /storage/data2        default.1                       booting             ro      nodrain   online
       cmseos54.fnal.gov (1095)    176   /storage/data1        default.0                       booting             ro      nodrain   online
       cmseos54.fnal.gov (1095)    177   /storage/data2        default.1                       booting             ro      nodrain   online
       cmseos55.fnal.gov (1095)    178   /storage/data1        default.2                       booting             ro      nodrain   online
       cmseos55.fnal.gov (1095)    179   /storage/data2        default.3                       booting             ro      nodrain   online
       cmseos57.fnal.gov (1095)    180   /storage/data1        default.2                       booting             ro      nodrain   online
       cmseos57.fnal.gov (1095)    181   /storage/data2        default.3                       booting             ro      nodrain   online
       cmseos58.fnal.gov (1095)    182   /storage/data1        default.0                       booting             ro      nodrain   online
       cmseos58.fnal.gov (1095)    183   /storage/data2        default.1                       booting             ro      nodrain   online
       cmseos59.fnal.gov (1095)    184   /storage/data1        default.2                       booting             ro      nodrain   online
       cmseos59.fnal.gov (1095)    185   /storage/data2        default.3                       booting             ro      nodrain   online
       cmseos60.fnal.gov (1095)    186   /storage/data1        default.0                       booting             ro      nodrain   online
       cmseos60.fnal.gov (1095)    187   /storage/data2        default.1                       booting             ro      nodrain   online
       cmseos61.fnal.gov (1095)    188   /storage/data1        default.2                       booting             ro      nodrain   online
       cmseos61.fnal.gov (1095)    189   /storage/data2        default.3                       booting             ro      nodrain   online
       cmseos62.fnal.gov (1095)    190   /storage/data1        default.0                       booting             ro      nodrain   online
       cmseos62.fnal.gov (1095)    191   /storage/data2        default.1                       booting             ro      nodrain   online
       cmseos63.fnal.gov (1095)    192   /storage/data1        default.2                       booting             ro      nodrain   online
       cmseos63.fnal.gov (1095)    193   /storage/data2        default.3                       booting             ro      nodrain   online
[root@cmssrv222 ~]# 

#4 Updated by Gerard Bernabeu Altayo over 3 years ago

So far not many failed:

[root@cmssrv222 ~]# eos fs ls -e

#...............................................................................................
#                   host #   id #     path #       boot # configstatus #      drain #... #errmsg
#...............................................................................................
       cmseos14.fnal.gov     52 ...e/data3  bootfailure             rw      nodrain  14 cannot clean SQLITE DB on local disk
        cmseos1.fnal.gov     64 ...e/data3  bootfailure             rw      nodrain  14 cannot clean SQLITE DB on local disk
[root@cmssrv222 ~]# 

Fixing this issue by forcing the metadata to be regenerated:

[root@cmseos14 ~]# service eos stop && mv /var/eos/md/*.sql /tmp/; service eos start
Stopping xrootd: fst                                       [  OK  ]

Starting xrootd as fst with -n fst -c /etc/xrd.cf.fst -l /var/log/eos/xrdlog.fst -b -Rdaemon
                                                           [  OK  ]
[root@cmseos14 ~]# 

I've done the same on cmseos1

#5 Updated by Gerard Bernabeu Altayo over 3 years ago

  • Status changed from New to Resolved

all FSTs are back online, closing.



Also available in: Atom PDF