Project

General

Profile

Support #10919

amove /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov

Added by Arthur Kreymer almost 4 years ago. Updated over 3 years ago.

Status:
New
Priority:
Normal
Start date:
11/12/2015
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Spent time:
Duration:

Description

We need to move /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov
This is due by Nov 20, this year.
This is part of the shutdown of the old Sun web servers.


Subtasks

Support #10921: ND DCS, DAQ, and Timing web pages all run on nusoft spaceNew

History

#1 Updated by Arthur Kreymer almost 4 years ago

What I think we might do right now is to rsync
/nusoft/app/web/htdoc/minos
to /web/sites/minos.fnal.gov/htdocs/nusoft

This way all present scripts can continue to push content,
we can update them later.

We first need to trim down controlroom/daqlogs

du -sm /nusoft/app/web/htdoc/minos/controlroom/daqlogs/*
112674 /nusoft/app/web/htdoc/minos/controlroom/daqlogs/FAR
22141 /nusoft/app/web/htdoc/minos/controlroom/daqlogs/NEAR

I have already made a static copy of /nusoft/app/web/cgi-bin/minos
to /web/sites/minos.fnal.gov/cgi-bin

fluffy.cgi and status work,
the other seem to depend on paths that will shift.

#2 Updated by Arthur Kreymer almost 4 years ago

Date: Fri, 13 Nov 2015 14:34:40 -0600
From: William F Badgett Jr <>

On what machines is "/web/sites" mounted?
----------------------
Just our SLF6 systems,
minos60-63, minos-data, minos-nearline, minos-slf6

Too ambitious to write to it directlry right now, I think,
Setting up an rsync from the old nusoft for now.

#3 Updated by Arthur Kreymer almost 4 years ago

Date: Fri, 13 Nov 2015 15:14:07 -0600
From: William F Badgett Jr <>

What will be the new write mechanism to the /web/sites directories?

Looks like everything is owned by "nobody:nobody" and not even readable
-------------
Date: Fri, 13 Nov 2015 15:19:09 -0600
From: Marc W Mengel <>

http://cd-docdb.fnal.gov/cgi-bin/RetrieveFile?docid=5375&filename=CWS%20Content%20Access%20Instructions.pdf&version=1
-------------------
For now, I plan to set up an rsync from the old area to the new one,
initially using my kerberos ticket, then soon a project keytab .
Most of our process can continue writing to the old areas
without modification in short term.

#4 Updated by Arthur Kreymer almost 4 years ago

The certificate seems to work for access to minos.fnal.gov .

KEYTAB=/opt/mindata/kt
export KRB5CCNAME=FILE:/tmp/krb5cc_minos_nusoftweb
KINIT=/usr/krb5/bin/kinit
${KINIT} -5 -A -k -t ${KEYTAB} ${PRINCIPAL}

I will start setting up an rsync.
First want to trim the content a bit,
mainly pre-shutdown controlroom/daqlogs/FAR/om/rootfiles ( 100 GB )

#5 Updated by Arthur Kreymer almost 4 years ago

    Preliminary scan of nusoft content, 
    finding large things that are not needed on the web.

     OVERALL

cd /nusoft/app/web/htdoc/minos
du -sm *  # omitting smaller areas

10052    beammonspill
57286    calibration
135159    controlroom
44310    OnlineMonitoring
9863    OnlineMonitoringFD

     OM

du -sm  OnlineMonitoring/OnlineMonitoringPlots/*
44292    OnlineMonitoring/OnlineMonitoringPlots/plots

2669 directories in plots, these have png images.

    CALIBRATION

48313    FdLinearityFitWorkspace

du -sm /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/*
...
246    /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/fdfit-2008-06
315    /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/fdfit-2008-07

du -sm /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/*
40    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-08
42    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-09
44    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-11

    DAQLOGS

du -sm NEAR/*
3    NEAR/daqMessage.txt
69    NEAR/dbu
1163    NEAR/dds
2449    NEAR/msglog
17771    NEAR/om
757    NEAR/timing
0    NEAR/TouchMe

1    NEAR/om/postscript
17734    NEAR/om/rootfiles
37    NEAR/om/summaries

 9009 root files typically 4 MB, Since Aug 2012

du -sm FAR/*
346    FAR/dbu
170    FAR/dds
2926    FAR/msglog
106158    FAR/om
1    FAR/test.html
3147    FAR/timing
0    FAR/TouchMe

  6880 root files up to 25 MB  Since Aug 2012

#6 Updated by Arthur Kreymer almost 4 years ago

I copied the large rootfiles directories from the web tree to the general data area,
creating a symlink for compatibility.
This will free up about 120 GB of web space.

Directories were
NUN=/nusoft/app/web/htdoc/minos/controlroom/daqlogs/NEAR/om/rootfiles
MDN=/minos/data/controlroom/daqlogs/NEAR/om/rootfiles

NUF=/nusoft/app/web/htdoc/minos/controlroom/daqlogs/FAR/om/rootfiles
MDF=/minos/data/controlroom/daqlogs/FAR/om/rootfiles

Copies were done around 14:05 for ND and 15:26 for FD.

The original rootfiles directories were renamed rootfiles-REMOVED.

I will leave these in place for a fallback for a day,
then remove them and copy /nusoft/app/web/htdoc/minos
to /web/sites/minos.fnal.gov/nusoft
for testing.

#7 Updated by Arthur Kreymer almost 4 years ago

Here is a summary of advice received last week from Andy Romero.

Access to /web/sites/* is via a special kernel kerberos ticket
that behaves differently than the customary kerberos ticket file.
Perhaps because the kernel is accessing the files, not the user processes.

There is a single kernel structure containing the current kerberos ticket
for each account.
In a shared account we use a project principal to access the web areas.
An interactive login with a forwarded user ticket can override this in the kernel,
possibily removing access to the web area.

The recommended solution is to do a few things :
  • Use an account/host not likely to be used for other work
    • At present, we are using minos@minos60 for controlroom updates
  • Do not forward user tickets when logging into that account
  • Set KRB5CCNAME in the .bash_profile for tha account being used
    • for minos60 , probably should do
      HOST=`hostname -s`
      [ "${HOST}" == 'minosgpvm04' ] && export KRB5CCNAME=FILE:/tmp/krb5cc_minos_web
      * set up a cron job to keep this ticket up to date several times a day
      
    • make a kinweb script in ~/bin
      #!/bin/sh
      export KRB5CCNAME=FILE:/tmp/krb5cc_minos_web
      KEYTAB=/opt/minos/minos-om.keytab
      PRINCIPAL=minos-wh-cr/minos/minos-om.fnal.gov@FNAL.GOV
      KINIT=/usr/krb5/bin/kinit
      ${KINIT} -5 -A  -k -t ${KEYTAB} ${PRINCIPAL}
      
    • run this via cron
      01 02,10,18 * * * ${HOME}/bin/kinweb
      

      I will to ask ECF/SSI to create /opt/minos on all servers

#8 Updated by Arthur Kreymer almost 4 years ago

Here's a summary of scans of various areas in nusoft/app/web/htdoc/minos

DATA > du -sm *
153    acclog
54    admin
37    beamdata
10052    beammonspill
19    bfield
8    bluwatch
1787    cal
57651    calibration
56    condor
19800    controlroom
2356    database
233    DataQuality
2295    dcache
12    farmgsum
98499    FDomroot
1    images
144    jira
635    maint
1    mcas
1    MinosBatch_AtAGlance
17921    NDomroot
202    OldMinosLEDataQualityWebsite
45761    OnlineMonitoring
10331    OnlineMonitoringFD
946    OnlineMonitoring.old
66    predator
1    prodmon
708    validation
1    vault
596    WebDocs

Searched for recent changes 11/24, found mainly

acclog
calibration/DriftInfo
calibration/NdLinearityFitWorkspace
calibration/FdLinearityFitWorkspace
database/topdb
dcache/poolsum
validation 
DataQuality
predator
beamdata
OnlineMonitoring

#9 Updated by Arthur Kreymer almost 4 years ago

  • Subject changed from move /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov to amove /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov

#10 Updated by Arthur Kreymer almost 4 years ago

I will add an executive summary of advice given so far on nusoft web migration
to https://cdcvs.fnal.gov/redmine/projects/admin/wiki/MNS-NUSOFT-WEB

Added rui to the watch list of this Issue.

#11 Updated by Arthur Kreymer over 3 years ago

I have started updating documentation at
https://cdcvs.fnal.gov/redmine/projects/admin/wiki/MNS-NUSOFT-WEB

Will continue after the holidays.



Also available in: Atom PDF