Project

General

Profile

News

System Outage w/ Lost data last week

Added by Marc Mengel 7 days ago

The system outage last Thursday was caused by a database server crash; to get things running they put up the last backup, which means Fermi Redmine is currently missing any changes from about 5:00PM Feb 5 through 12:00 noon Feb 6.

We are working on recovering the missing data;It appears to be
73 wiki pages
43 issue updates
18 attachment uploads

-- Update:

so far I've recovered the changes for issue trackers, and project wiki's 'A' through 'N'
I have another 30 wiki pages to check, and then 11 attachments to re-attach...

Assorted outages Thursday Dec 21

Added by Marc Mengel about 2 years ago

The system (ccdcvs) that we run on, and the database servers we use will be down various times Thursday morning between 7am and noon.


ccdcvsvm System/Process Owners,

This is a reminder that it's patching time again for ccdcvsvm, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 12/21/17, at 7:00AM CDT.

Thanks,

Linux Server Support Group


There is a planned downtime scheduled for tomorrow, Thursday, Dec. 21; from 8 a.m. to noon Central Time.

As part of this downtime, ECF will be performing kernel updates and reboots on a variety of systems, including servers ifdb04, ifdb05, and ifdb06.

Please expect downtime for the following databases:

Hosted on ifdb04 (ifdbdev):

  • nova_hardware_dev
  • nova_hardware_int
  • nova_hardware_drop
  • nova_dev
  • nova_ashriver_dev
  • nova_ecl_dev
  • bamon_dev
  • microboone_dev
  • artdaq_db01_dev
  • ifb_dev
  • larsoft_dev
  • ci_art_dev
  • ci_g2_dev
  • ci_mu2e_dev
  • lariat_dev
  • dune35t_dev
  • pdune_hardware_dev
  • dune_colldb_dev
  • lariat_dqm_dev
  • mu2e_ucon_dev
  • gm2_conditions_dev
  • artdaq_db02_dev
  • mnvcon_int
  • pomsdev
  • pdunesp_dev
  • icarus_hardware_dev

Hosted on ifdb05 (ifdbprod, ifdbprod2):

  • nova_prod
  • nova_ecl_prd
  • nova_ashriver_prd
  • novapro_ecl_prd
  • nova_hardware
  • bamon_prd
  • larsoft_prd
  • ci_nova_prd
  • ci_art_prd
  • ci_gwms_prd
  • ci_minerva_prd
  • ci_genie_prd
  • ci_g2_prd
  • ci_mu2e_prd
  • dune35t_prod
  • pdune_hardware_prd
  • dune_colldb_prd
  • lariat_prd
  • lariat_dqm_prd
  • lariatcalib_prod
  • microboone_prod
  • hootgibson_prod
  • mu2e_hardware_prd
  • pomsprd
  • mu2e_ucon_prod
  • mnvcon_int
  • mnvcon_prd
  • pdunesp_prod
  • gm2_online_prod
  • gm2_conditions_prod
  • icarus_prd

Hosted on ifdb06 (ifdbrep):

  • nova_prod
  • nova_ecl_prd
  • nova_ashriver_prd
  • novapro_ecl_prd
  • nova_hardware
  • nova_prod (replication from novadcs-far-logger to ifdb06)
  • nova_prod (replication from novadaq-near-db-01 to novadaq-near-db-02, to ifdb06)
  • nova_prod (replication from novadaq-far-db-03 to novadaq-far-db-04, to ifdb06)
  • nova_prod (replication from novadcs-near-logger-101)
  • pdunesp_prod
  • gm2_online_prod
  • gm2_conditions_prod *

Databases replicated from ubdaq-prod-smc to uboonedaq-seb-10 to ifdb06:

  • procdb
  • procdb_sn
  • slowmoncon_archive
  • slowmoncon
  • slowmoncon_alarm
  • slowmoncon_log
  • runconfdb
  • eandatests
  • mrttests
  • rctests
  • rctestsdev
  • rctestskazu
  • testprocdb
  • testrunconfdb

Please distribute this message to those in your organization who may be impacted.

Thank you,

Olga.

Third Thursday AM patching reboot

Added by Marc Mengel over 3 years ago

Our Redmine host and our database are being patched:

This is a reminder that it's patching time again for ccdcvsvm, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 10/20/16, at 7:00AM CDT.

This is a reminder that it's patching time again for fnalpgsprd, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 10/20/16, at 06:30AM CDT.

Thanks,

Linux Server Support Group

(1-10/41)

Also available in: Atom