Project

General

Profile

Support #22335

Minos Production Proxy needs renewal

Added by Arthur Kreymer 8 months ago. Updated 7 months ago.

Status:
Resolved
Priority:
Normal
Start date:
04/08/2019
Due date:
04/22/2019
% Done:

100%

Estimated time:
5.00 h
Duration: 15

Description

We have learned that the Minos production proxy needs to be renewed soon.

I will post the email we received separately.

History

#1 Updated by Arthur Kreymer 8 months ago

  • Assignee set to Thomas Carroll

Date: Mon, 8 Apr 2019 17:23:31 +0000
From: Shreyas Bhat <>
To: Arthur E Kreymer <>, Adam P Schreckenberger <>
Subject: minos Managed Proxy Service Cert Expiring Soon

Dear all,

Within the next month, the managed proxies for minos will be expiring.  In order to ensure that you have no break in service
with these proxies, we will need to update them.

Because the OSG Certificate Authority shut down last year, we will be issuing new certificates signed by the InCommon CA.  These
certificates will have a different DN
subject than before, and thus will need to be reregistered.  The plan to deploy these will be as follows:

1. Confirm DN and registration person with experiment (this email - PLEASE CONFIRM!)
2. Get new cert and start pushing it to /<current_destination_directory>/test/<account_name>/<account>.<role>.proxy (USDC does)
3. Add cert to SAM (USDC does)
5. Experiment testing
6. Experiment go live (simply change the directory the test proxy is pushed to the normal one)

We want to have this wrapped up by the end of the week if possible, since the transition itself is pretty quick and testing can
begin as soon as tomorrow.

Currently, the managed proxy service certificate(s) for minos is registered to Art.  Please confirm that this is still OK,
and we'll issue a new certificate, register the cert to that same person in FERRY and SAM, and let you know you can begin
testing.
Note that due to the new CA, there will be a slight CN subject change (something like
"/DC=org/DC=opensciencegrid/O=Open Science Grid/OU=Services/CN=geniepro/geniegpvm01.fnal.gov" will become
"/DC=org/DC=incommon/C=US/ST=IL/L=Batavia/O=Fermi Research Alliance/OU=Fermilab/CN=geniepro-geniegpvm01.fnal.gov").

Please let me know if you have any questions.

Thanks very much,
Shreyas

#2 Updated by Arthur Kreymer 8 months ago

There is a short term need to get feedback on beam quality for NOvA,
using the Minos Near detector, operated by Minerva

There have been a series of issues running keepup processing,
including but not limited to dCache issues and database issues.
https://cdcvs.fnal.gov/redmine/issues/22262

This work probably needs to be finished before the new proxies can be tested.

Robert Hatcher - should the new proxies be registered to you ?
You are still a long term employee at Fermilab, and I retired 2 years ago.

#3 Updated by Shreyas Bhat 8 months ago

I've issued the new certificate. Once you let me know who the cert should be registered to, I can do that and you can start testing when you're ready.

#4 Updated by Arthur Kreymer 7 months ago

. ./jobsub.sh

export X509_USER_PROXY=/opt/minospro/test/minospro/minospro.Production.proxy
FAST=' --expected-lifetime=1h --memory=100MB --disk=3GB'
PROBE=/grid/fermiapp/minos/scripts/probe

jobsub_q
11 running AMBROSIA

jobsub_submit -g ${FAST} file://${PROBE}

JID=
TLD=//var/tmp/`whoami`/${JID}

jobsub_q
...
minospro 04/12 10:32 0+00:02:00 R 0 0.0 probe_20190412_103257_3849486_0_1_wrap.sh

The job ran over 8 minutes, but should have taken only a few seconds.

At 10:47, there are no *.out or *.err files :

jobsub_fetchlog --list | grep ${JID}
Fri Apr 12 10:40:47 2019

jobsub_fetchlog --jobid=${JID} --dest-dir=${TLD} --role=Production

ls -l ${TLD}

total 36
rwxrwxr-x 1 minospro e875 12427 Apr 12 10:32 probe
-rw-r--r-
1 minospro e875 2256 Apr 12 10:32 probe_20190412_103257_3849486_0_1_.cmd
rw-r--r- 1 minospro e875 5408 Apr 12 10:38 probe_20190412_103257_3849486_0_1_.log
-rwxr-xr-x 1 minospro e875 6592 Apr 12 10:32 probe_20190412_103257_3849486_0_1_wrap.sh

#5 Updated by Arthur Kreymer 7 months ago

Shreyas - Please look to see what happened to the output of

#6 Updated by Arthur Kreymer 7 months ago

Never mind, the output files finally showed up around 11:10 or so.
I fetched fresh copies of the job files.

The PROXY being held by the job is reported as

PROXY /storage/local/data1/condor/execute/dir_31355/x509cc_minospro_Production
identity : /DC=org/DC=incommon/C=US/ST=IL/L=Batavia/O=Fermi Research Alliance/OU=Fermilab/CN=minospro-minos27.fnal.gov/CN=1837990350/CN=334418435/CN=974466069

Before moving the new certs to production,
we should verify that a reconstruction job an run and return output files.

#7 Updated by Arthur Kreymer 7 months ago

  • % Done changed from 0 to 50

#8 Updated by Shreyas Bhat 7 months ago

Sounds good to me. It'll only take a minute to deploy the certs to production, so go ahead and run any other tests and let me know when you're ready. If we run into the cigetcert issues in production, I can always push the new cert early.

#9 Updated by Arthur Kreymer 7 months ago

  • % Done changed from 50 to 100
  • Status changed from New to Resolved

I am marking this Issue as Resolved.
It appears that the new proxies were deployed around April 25.

#10 Updated by Shreyas Bhat 7 months ago

Correct - the new MINOS proxies were deployed earlier than anticipated; the deployment time was April 24, around 2 p.m.



Also available in: Atom PDF