Project

General

Profile

Maint-20121115

Last Modified {{last_modified}}

  • minos25,50-54,minos-db1 will be rebooted with new kernels
    • 2.6.18-308.16.1.el5 Release Date: Oct 16, 2012
  • GPCF maintenance all morning, affecting minos25, minos-slf4, minos-slf5
  • DCache and Enstore updates
    • Down 08:00 to 12:00 CDT
    • Affects :
      • PNFS
      • DCache
      • Enstore
  • Fermigrid GPGrid condor upgrade
    • Jobs will start draining Wed Nov 14
  • Fermicloud 09:00-12:00
    • Affects GridFTP servers, and minos-admin
  • Bluearc firmware update and failback to primary server
    • Bluearc should remain up, using alternate servers
    • There may be short pauses as services migrate

http://fefweb.fnal.gov/mediawiki/index.php/IF,_EAG,_GPGrid,_GPCF_Downtime_-_15_Nov_2012

SERVICE DOWN UP COMMENT
Servers 08:40 11:01 see details in next table
minos-db1 09:11 09:50 INC000000338938 rebooted 09:13, mysql was not restarted, DBA started manually
Enstore/Dcache 08:00 12:15 Minos saw no outages
GPGrid 08:30 11:00 draining 16:00 11/14, probe jobs started running 10:57 11/15
Fermicloud
Bluearc 06:00 06:56 No failures, slow access observed at 06:11 and 06:15
SERVER DOWN UP COMMENT
minos25 08:40 10:08 part of GPCF
minos50 09:45 10:22 mount /pnfs/minos
minos51 09:45 11:01 mount /pnfs/minos
minos52 09:45 09:48 mount /pnfs/minos
minos53 09:45 09:48 mount /pnfs/minos
minos54 09:45 09:48 mount /pnfs/minos
minos-slf4 08:45 10:22 mount /pnfs/minos
minos-slf5 08:38 10:08

Issues after planned maintenance - none

  • Expected :
    • remount /pnfs/minos
    • restart predator
    • restart roundup
  • Pending
  • Fixed
  • mount /pnfs/minos on minos50-53 - by hand
  • INC000000338938 start Minos mysql database - RESOLVED
  • INC000000339128 /minos/data and /minos/app readonly on minos27 - RESOLVED
    • stopped lockclean and roundup so file systems could be remounted around 14:30 CDT
    • restarted lockclean and roundup