Fermi Redmine

I believe we now have the data lost from the crash last Thursday all put back. If you have data missing still from Thursday, contact me: mengel@fnal.gov

The system outage last Thursday was caused by a database server crash; to get things running they put up the last backup, which means Fermi Redmine is currently missing any changes from about 5:00PM Feb 5 through 12:00 noon Feb 6.

We are working on recovering the missing data;It appears to be
73 wiki pages
43 issue updates
18 attachment uploads

-- Update:

so far I've recovered the changes for issue trackers, and project wiki's 'A' through 'N'
I have another 30 wiki pages to check, and then 11 attachments to re-attach...

We have upgraded Redmine to the latest 3.4 release.

Update: The database upgrade for Redmine is complete as of 10:20 CST

The database behind Redmine is being Upgraded on Wednesday Jan 16

The Change on Service Now says the outage will be at 09:00 - 16:00

We've updated the system to Redmine 3.4.3

The system (ccdcvs) that we run on, and the database servers we use will be down various times Thursday morning between 7am and noon.

ccdcvsvm System/Process Owners,

This is a reminder that it's patching time again for ccdcvsvm, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 12/21/17, at 7:00AM CDT.

Thanks,

Linux Server Support Group

There is a planned downtime scheduled for tomorrow, Thursday, Dec. 21; from 8 a.m. to noon Central Time.

As part of this downtime, ECF will be performing kernel updates and reboots on a variety of systems, including servers ifdb04, ifdb05, and ifdb06.

Please expect downtime for the following databases:

Hosted on ifdb04 (ifdbdev):

• nova_hardware_dev
• nova_hardware_int
• nova_hardware_drop
• nova_dev
• nova_ashriver_dev
• nova_ecl_dev
• bamon_dev
• microboone_dev
• artdaq_db01_dev
• ifb_dev
• larsoft_dev
• ci_art_dev
• ci_g2_dev
• ci_mu2e_dev
• lariat_dev
• dune35t_dev
• pdune_hardware_dev
• dune_colldb_dev
• lariat_dqm_dev
• mu2e_ucon_dev
• gm2_conditions_dev
• artdaq_db02_dev
• mnvcon_int
• pomsdev
• pdunesp_dev
• icarus_hardware_dev

Hosted on ifdb05 (ifdbprod, ifdbprod2):

• nova_prod
• nova_ecl_prd
• nova_ashriver_prd
• novapro_ecl_prd
• nova_hardware
• bamon_prd
• larsoft_prd
• ci_nova_prd
• ci_art_prd
• ci_gwms_prd
• ci_minerva_prd
• ci_genie_prd
• ci_g2_prd
• ci_mu2e_prd
• dune35t_prod
• pdune_hardware_prd
• dune_colldb_prd
• lariat_prd
• lariat_dqm_prd
• lariatcalib_prod
• microboone_prod
• hootgibson_prod
• mu2e_hardware_prd
• pomsprd
• mu2e_ucon_prod
• mnvcon_int
• mnvcon_prd
• pdunesp_prod
• gm2_online_prod
• gm2_conditions_prod
• icarus_prd

Hosted on ifdb06 (ifdbrep):

• nova_prod
• nova_ecl_prd
• nova_ashriver_prd
• novapro_ecl_prd
• nova_hardware
• nova_prod (replication from novadcs-far-logger to ifdb06)
• nova_prod (replication from novadaq-near-db-01 to novadaq-near-db-02, to ifdb06)
• nova_prod (replication from novadaq-far-db-03 to novadaq-far-db-04, to ifdb06)
• nova_prod (replication from novadcs-near-logger-101)
• pdunesp_prod
• gm2_online_prod
• gm2_conditions_prod *

Databases replicated from ubdaq-prod-smc to uboonedaq-seb-10 to ifdb06:

• procdb
• procdb_sn
• slowmoncon_archive
• slowmoncon
• slowmoncon_alarm
• slowmoncon_log
• runconfdb
• eandatests
• mrttests
• rctests
• rctestsdev
• rctestskazu
• testprocdb
• testrunconfdb

Please distribute this message to those in your organization who may be impacted.

Thank you,

Olga.

The database server behind Fermi Redmine, and the system itself will be rebooted for OS patches, etc. to take effect Thursday Morning at 06:30 and 07:00. Both outages should be short, maybe a few minutes.

We may have a reboot 07:00 AM Thursday to put in the DirtyCOW security patch.

Our Redmine host and our database are being patched:

This is a reminder that it's patching time again for ccdcvsvm, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 10/20/16, at 7:00AM CDT.

This is a reminder that it's patching time again for fnalpgsprd, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 10/20/16, at 06:30AM CDT.

Thanks,

Linux Server Support Group

I have started a "User Discussion" forum in the Fermi Redmine project:
https://cdcvs.fnal.gov/redmine/projects/fermi-redmine/boards
for discussion, etc.