Fermilab Redmine

The system (ccdcvs) that we run on, and the database servers we use will be down various times Thursday morning between 7am and noon.

ccdcvsvm System/Process Owners,

This is a reminder that it's patching time again for ccdcvsvm, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 12/21/17, at 7:00AM CDT.

Thanks,

Linux Server Support Group

There is a planned downtime scheduled for tomorrow, Thursday, Dec. 21; from 8 a.m. to noon Central Time.

As part of this downtime, ECF will be performing kernel updates and reboots on a variety of systems, including servers ifdb04, ifdb05, and ifdb06.

Please expect downtime for the following databases:

Hosted on ifdb04 (ifdbdev):

• nova_hardware_dev
• nova_hardware_int
• nova_hardware_drop
• nova_dev
• nova_ashriver_dev
• nova_ecl_dev
• bamon_dev
• microboone_dev
• artdaq_db01_dev
• ifb_dev
• larsoft_dev
• ci_art_dev
• ci_g2_dev
• ci_mu2e_dev
• lariat_dev
• dune35t_dev
• pdune_hardware_dev
• dune_colldb_dev
• lariat_dqm_dev
• mu2e_ucon_dev
• gm2_conditions_dev
• artdaq_db02_dev
• mnvcon_int
• pomsdev
• pdunesp_dev
• icarus_hardware_dev

Hosted on ifdb05 (ifdbprod, ifdbprod2):

• nova_prod
• nova_ecl_prd
• nova_ashriver_prd
• novapro_ecl_prd
• nova_hardware
• bamon_prd
• larsoft_prd
• ci_nova_prd
• ci_art_prd
• ci_gwms_prd
• ci_minerva_prd
• ci_genie_prd
• ci_g2_prd
• ci_mu2e_prd
• dune35t_prod
• pdune_hardware_prd
• dune_colldb_prd
• lariat_prd
• lariat_dqm_prd
• lariatcalib_prod
• microboone_prod
• hootgibson_prod
• mu2e_hardware_prd
• pomsprd
• mu2e_ucon_prod
• mnvcon_int
• mnvcon_prd
• pdunesp_prod
• gm2_online_prod
• gm2_conditions_prod
• icarus_prd

Hosted on ifdb06 (ifdbrep):

• nova_prod
• nova_ecl_prd
• nova_ashriver_prd
• novapro_ecl_prd
• nova_hardware
• nova_prod (replication from novadcs-far-logger to ifdb06)
• nova_prod (replication from novadaq-near-db-01 to novadaq-near-db-02, to ifdb06)
• nova_prod (replication from novadaq-far-db-03 to novadaq-far-db-04, to ifdb06)
• nova_prod (replication from novadcs-near-logger-101)
• pdunesp_prod
• gm2_online_prod
• gm2_conditions_prod *

Databases replicated from ubdaq-prod-smc to uboonedaq-seb-10 to ifdb06:

• procdb
• procdb_sn
• slowmoncon_archive
• slowmoncon
• slowmoncon_alarm
• slowmoncon_log
• runconfdb
• eandatests
• mrttests
• rctests
• rctestsdev
• rctestskazu
• testprocdb
• testrunconfdb

Please distribute this message to those in your organization who may be impacted.

Thank you,

Olga.

The database server behind Fermi Redmine, and the system itself will be rebooted for OS patches, etc. to take effect Thursday Morning at 06:30 and 07:00. Both outages should be short, maybe a few minutes.

The Elog was restarted to fix bugs and implements suggested features.

Power outage dates for moving power loads back to the Master SubStation.

A new version of the elog was released 11/22.

- Still recovering from the effects of the partial power outage 2 weeks ago, resulting in sprinkler discharge in FD server room. Experts from Fwermilab & Caltech sent yesterday to start energizing system. Power up started this morning.
ND running smoothly.

When the first 1.2s $2A ramp was tried, MECAR ramped to around 4000A and then tripped off with a No Current Halt. We determined that the PS were not ramping up as they should have.

I remembered that some of the Real Time Events in MECAR have negative time offsets. Upon checking I realized that the Bend Bus Anticipation Event Offset was set to -50mSec. Because the initial DC section of the ramp has been shortened from 80mSec to 42mSec this means the this event doesn't even occur. This is bad.

As a test I shortened the Bend Bus Anticipation Event Offset to -40mSec. This seems to be working.

The quad bus's Anticipation Event Offset had a similar but opposite problem. The Quad Bus Anticipation Event Offsets were set to 69mSec. Because the PS are ramping earlier this was causing the Quad Bus Tier 0 to not turn on in time. I have set the Horz and Vert Anticipation Event Offsets to 10mSec. This seems to be working.

We may have a reboot 07:00 AM Thursday to put in the DirtyCOW security patch.

Our Redmine host and our database are being patched:

This is a reminder that it's patching time again for ccdcvsvm, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 10/20/16, at 7:00AM CDT.

This is a reminder that it's patching time again for fnalpgsprd, according to the agreed upon schedule.
Only a reboot is required. Downtime will be <30 minutes.
The system will be rebooted on Thursday, 10/20/16, at 06:30AM CDT.

Thanks,

Linux Server Support Group

Newest AD Ops Schedule¶

Beam to BNB off November 03, 2016 until ~ 15:30 today.

Plan is to have beam to the MI dump on October 28th and to start low energy target scans on 31st of October.