Fermi Redmine

Dear all,

It's patching time again for CDPGSPRD. The system will be rebooted Wednesday, 10/21/20, at 8:00AM CDT.
Downtime will be less than 30 minutes. Please expect interruption accessing the following databases:

fermilab_mulc_prd
edoffice_prd
cdcvs_redmine_prd
rboard_prd
ferry_prd
indico
ita_data_manager_prd
drimagedb
g4validation
validationdb
ad_db
des_publications_prd
muondb_prd
ppd_cal_prd
accelai_meta_prod
rextracking_prd
jabber1
lxr_cdf

The database host for Redmine will be getting system patches.

I believe we now have the data lost from the crash last Thursday all put back. If you have data missing still from Thursday, contact me: mengel@fnal.gov

The system outage last Thursday was caused by a database server crash; to get things running they put up the last backup, which means Fermi Redmine is currently missing any changes from about 5:00PM Feb 5 through 12:00 noon Feb 6.

We are working on recovering the missing data;It appears to be
73 wiki pages
43 issue updates
18 attachment uploads

-- Update:

so far I've recovered the changes for issue trackers, and project wiki's 'A' through 'N'
I have another 30 wiki pages to check, and then 11 attachments to re-attach...

We have upgraded Redmine to the latest 3.4 release.

Update: The database upgrade for Redmine is complete as of 10:20 CST

The database behind Redmine is being Upgraded on Wednesday Jan 16

The Change on Service Now says the outage will be at 09:00 - 16:00

We've updated the system to Redmine 3.4.3

The system (ccdcvs) that we run on, and the database servers we use will be down various times Thursday morning between 7am and noon.

ccdcvsvm System/Process Owners,

This is a reminder that it's patching time again for ccdcvsvm, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 12/21/17, at 7:00AM CDT.

Thanks,

Linux Server Support Group

There is a planned downtime scheduled for tomorrow, Thursday, Dec. 21; from 8 a.m. to noon Central Time.

As part of this downtime, ECF will be performing kernel updates and reboots on a variety of systems, including servers ifdb04, ifdb05, and ifdb06.

Please expect downtime for the following databases:

Hosted on ifdb04 (ifdbdev):

• nova_hardware_dev
• nova_hardware_int
• nova_hardware_drop
• nova_dev
• nova_ashriver_dev
• nova_ecl_dev
• bamon_dev
• microboone_dev
• artdaq_db01_dev
• ifb_dev
• larsoft_dev
• ci_art_dev
• ci_g2_dev
• ci_mu2e_dev
• lariat_dev
• dune35t_dev
• pdune_hardware_dev
• dune_colldb_dev
• lariat_dqm_dev
• mu2e_ucon_dev
• gm2_conditions_dev
• artdaq_db02_dev
• mnvcon_int
• pomsdev
• pdunesp_dev
• icarus_hardware_dev

Hosted on ifdb05 (ifdbprod, ifdbprod2):

• nova_prod
• nova_ecl_prd
• nova_ashriver_prd
• novapro_ecl_prd
• nova_hardware
• bamon_prd
• larsoft_prd
• ci_nova_prd
• ci_art_prd
• ci_gwms_prd
• ci_minerva_prd
• ci_genie_prd
• ci_g2_prd
• ci_mu2e_prd
• dune35t_prod
• pdune_hardware_prd
• dune_colldb_prd
• lariat_prd
• lariat_dqm_prd
• lariatcalib_prod
• microboone_prod
• hootgibson_prod
• mu2e_hardware_prd
• pomsprd
• mu2e_ucon_prod
• mnvcon_int
• mnvcon_prd
• pdunesp_prod
• gm2_online_prod
• gm2_conditions_prod
• icarus_prd

Hosted on ifdb06 (ifdbrep):

• nova_prod
• nova_ecl_prd
• nova_ashriver_prd
• novapro_ecl_prd
• nova_hardware
• nova_prod (replication from novadcs-far-logger to ifdb06)
• nova_prod (replication from novadaq-near-db-01 to novadaq-near-db-02, to ifdb06)
• nova_prod (replication from novadaq-far-db-03 to novadaq-far-db-04, to ifdb06)
• nova_prod (replication from novadcs-near-logger-101)
• pdunesp_prod
• gm2_online_prod
• gm2_conditions_prod *

Databases replicated from ubdaq-prod-smc to uboonedaq-seb-10 to ifdb06:

• procdb
• procdb_sn
• slowmoncon_archive
• slowmoncon
• slowmoncon_alarm
• slowmoncon_log
• runconfdb
• eandatests
• mrttests
• rctests
• rctestsdev
• rctestskazu
• testprocdb
• testrunconfdb

Please distribute this message to those in your organization who may be impacted.

Thank you,

Olga.

The database server behind Fermi Redmine, and the system itself will be rebooted for OS patches, etc. to take effect Thursday Morning at 06:30 and 07:00. Both outages should be short, maybe a few minutes.

We may have a reboot 07:00 AM Thursday to put in the DirtyCOW security patch.