Project

General

Profile

Scheduled maintenance 2012 May 1 to 12

updated 2012-05-02 20:51 UTC

  • Bluearc firmware upgrade - COMPLETE 05:47
    • /minos/* and /grid/* file systems
    • Fermigrid
    • DocDB
    • ECL
    • Redmine including project, wiki and bug-tracking sites
    • CVS, Git and Subversion repositories
    • Many Fermilab web sites
    • File transfer protocol service using ftp.fnal.gov
    • ups and upd software distribution
Down May 1 19:00 CDT
Up May 2 06:00 CDT /grid/data and /minos/data up around midnight
/minos/app up around 04:10
DocDB, ECL, Redmine, CVS, fermilinux website all OK
Maintenance declared complete at 05:47
  • minos25 move from GCC to FCC and expand CPU and memory
    • This is the Minos condor master. Condor will be stopped.
    • Scheduled Wednesday morning
SERVICE DOWN UP Notes
GlideinWMS May 1 14:00 May 2 ... pending move of minos25
Condor May 1 17:00 May 2 ... pending move of minos25
MINOS25 May 2 09:46 May 2 11:44 initial booted with time 16:24 corrected at 11:33
  • All Minos servers will be booted after the Bluarc maintenance.
    • To get clean Bluearc mounts
    • To install new kernels, for security
    • kernel 2.6.18-308.4.1.el5
SYSTEM UP Notes
minos25 11:33 CDT VM moved from GCC to FCC. See 4 CPU, 12 GB
minos27 12:00 CDT not on FEF list see INC000000245809
minos50 10:48 CDT uptime claims 10:08, booted at 10:48
minos51 11:23 CDT fsck delayed reboot, uptime reports 10:08
minos52 10:08 CDT
minos53 10:08 CDT
minos54 10:08 CDT
minos-sam02 10:08 CDT dbservers were running
minos-sam04 10:08 CDT dbservers were running
minos-mysql2 11:03 CDT Started mysql around 11:06
minos-slf5 10:08 CDT

Issues after planned maintenance

  • Expected :
    • FEF remount grid areas as necessary
    • MINOS - restart Predator, Roundup, Mcimport
  • Fixed :
    • Condor restarted cleanly by dbox around 18:00
  • Pending :