Project

General

Profile

Support #25049

test fnalmariadbpord server

Added by Arthur Kreymer about 2 months ago. Updated 6 days ago.

Status:
Work in progress
Priority:
High
Start date:
10/05/2020
Due date:
11/30/2020
% Done:

90%

Estimated time:
20.00 h
Duration: 57

Description

Test the fnalmariadbprd service which will replace mariadb-prd1/2/3
as requested in RITM1008759

The test server is fnalmariadbdev

Testing procedures are documented in
https://cdcvs.fnal.gov/redmine/projects/database/wiki/MINOSDBTEST

Minos_DB_status_053017.pdf (606 KB) Minos_DB_status_053017.pdf Arthur Kreymer, 05/31/2017 09:09 AM

Related issues

Copied from MINOS - Support #16021: test mariadb serverClosed03/29/201706/15/2017

History

#1 Updated by Arthur Kreymer about 2 months ago

#2 Updated by Arthur Kreymer about 2 months ago

PLAN

TASK DONE COMMENT
Replicate 08/28 cloned by DBAs, see RITM1008759
Monitoring 10/04 http://minos.fnal.gov/database/topdb/fnalmariaprd/
Accounts 10/04 tested reader, writer
DBU test write 10/05 Ran on minos-slf6, and minos-data under singularity
Minos analysis
Minerva analysis 09/24 see INC000001105278
connection limits by DBAs
file descriptor limits by DBAs

Redhat 6 loses support at the end of November 2020.

The Minos database is on a Galera cluster of three Redhat 6 nodes mysql-prd1/2/3 running Redhat 6, configured wihth an InnoDB backend.

The Minos database will run on a single Redhat 8 node fnalmysqldbprd running a MyISAM backend for improved performance.

Future system upgrades will be done by swapping service to fnalmysqldbdev.
This should be transparent to users, who continue to connect to alias minos-db1.

#3 Updated by Arthur Kreymer about 2 months ago

  • % Done changed from 20 to 40

I ran dbu tests against the new server.
I did this both on minos-data, under singularity,
and on the old minos-sl6 host.

I used the standard test procedures,
but changed the LOON release
from
LOON="-r R3.01.00 --32bit"
to
LOON="-r R3.11"
as there were problems setting up the older release.

And of course I set NEW=fnalmariadbprd

The log file and generated sam metadata look consistent with what is seen connecting to minos-db1.

#4 Updated by Arthur Kreymer about 2 months ago

To: minos-users@fnal.gov

Subj: Please test the new database server

The Minos offline database is being moved to new servers.

Preliminary tests look good, see
https://cdcvs.fnal.gov/redmine/issues/25049

Please test your analysis code with the new servers.

Change minos-db1 to fnalmariadbprd in the database connection, like
export ENV_TSQL_URL='mysql:odbc://fnalmariadbprd.fnal.gov/offline'

Please update the Issue with your results.
Thanks !

#5 Updated by Arthur Kreymer about 1 month ago

  • % Done changed from 40 to 90

We updated RITM1008759
I will close this Testing Issue when the server is migrated


At the 2020 Oct 19 Minos All Analysis meeting,
Minos approved redirecting the minos-db1 alias to the new fnalmariadbprd server.
This should be transparent to the users.

Please schedule this at your next convenience.

Please keep a permanent full backup of the database from the old server.

Please keep the old servers online for a short time after the transition, in case some issue arises.

#6 Updated by Arthur Kreymer about 1 month ago

  • % Done changed from 90 to 70

I have tried to see what would be needed to change the Minos setup to specify a port number.

All the old version of ${MINOS_SETUP_DIR}/setup_minossoft_*.sh set ENV_TSQL_URL inline.

The script we currently use, setup_minossoft_FNALU.sh, instead inherits this from elsewhere.
It invokes the adjust_tsql.sh script, which modifies but does not set the server.

After some serious digging, I see that ENV_TSQL_URL
seems to set in the Minos UPS product
minos_config v0.2 -q default.

The file to edit would seem to be
/grid/fermiapp/minos/minossoft_srt_64bit/minos_ups_products/minos_config/v02/NULL/default/ups/minos_config.table

I did find an issue in adjust_tsql.sh.
The add_extra_db function actively accesses the database at the default port.
This would have to be changed if we use a non-default port.

Before we change the port,
I would like to hear directly from someone in Computer Security that this is absolutely required.

My previous security training at Fermilab repeatedly stated something like 'There is no Security in Obscurity'
All open ports will be scanned, so changing port numbers does not improve security.

#7 Updated by Arthur Kreymer about 1 month ago

From: Olga Vlasova <>
To: Rebecca Bensinger <>
CC: Arthur S Lee <>, John L Galvan <>, Svetlana G Lebedeva <>, Arthur E Kreymer <>, Mitchell Renfer <>
Subject: Security Concern
Date: Tue, 20 Oct 2020 16:14:26 +0000

Hello Rebecca,

We support Minos MariaDB databases and historically Minos collaboration
have been using default port 3306. Blocking offsite nodes would remove access
from the Open Science Grid, so port 3306 needs to stay open for offsite.

We are ready to migrate Minos production databases from
mariadb-prd1,-prd2,-prd3 nodes to fnalmariadbprd
and development ones to fnalmariadbdev.
Art agreed to restrict permissions to the offsite users to read-only.

Please let us know if we will be able to use the default port
and keep it open for the offsite users.

Thanks,
Olga.

#8 Updated by Arthur Kreymer about 1 month ago

Date: Thu, 22 Oct 2020 12:39:31 -0500
Subject: Re: Security Concern
From: Arthur Kreymer <>
To: Olga Vlasova <>
Cc: Rebecca Bensinger <>, Arthur S Lee <>, John L Galvan <>, Svetlana G Lebedeva <>, Arthur E Kreymer <>, Mitchell Renfer <>, Robert W Hatcher <>

Minos rarely writes to the conditions database, and then only from a
special database account.
There is a chance that Minos will not need to perform further writes,
as it moves into Data Preservation mode.

I suggest that we further reduce exposure of the Minos database
by allowing writes only from a single node, minosdatagpvm03.

We could consider shutting even that channel down,
with the option to allow writes briefly by special request, if needed.

#9 Updated by Arthur Kreymer 6 days ago

  • % Done changed from 70 to 90
  • Due date changed from 11/01/2020 to 11/30/2020

minos-db1 was redirected to fnalmariadbprd around noon on Nov 18.
A request for testing was sent to the minos-users mailing list.

The old servers will be shut down Nov 30.

I used the ~mindata/conndb script to test that reader and writer accounts connect to the new server.

I ran the traditional dbu update test from mindata@minosgpvm03, this seemed to run OK.

Also available in: Atom PDF