Project

General

Profile

Support #10919

move /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov

Added by Arthur Kreymer over 4 years ago. Updated 10 days ago.

Status:
Work in progress
Priority:
Normal
Start date:
11/12/2015
Due date:
% Done:

0%

Estimated time:
40.00 h
Spent time:
Duration:

Description

We need to move /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov
This is due by Nov 20, this year.
This is part of the shutdown of the old Sun web servers.


Subtasks

Support #10921: ND DCS, DAQ, and Timing web pages all run on nusoft spaceNew

History

#1 Updated by Arthur Kreymer over 4 years ago

What I think we might do right now is to rsync
/nusoft/app/web/htdoc/minos
to /web/sites/minos.fnal.gov/htdocs/nusoft

This way all present scripts can continue to push content,
we can update them later.

We first need to trim down controlroom/daqlogs

du -sm /nusoft/app/web/htdoc/minos/controlroom/daqlogs/*
112674 /nusoft/app/web/htdoc/minos/controlroom/daqlogs/FAR
22141 /nusoft/app/web/htdoc/minos/controlroom/daqlogs/NEAR

I have already made a static copy of /nusoft/app/web/cgi-bin/minos
to /web/sites/minos.fnal.gov/cgi-bin

fluffy.cgi and status work,
the other seem to depend on paths that will shift.

#2 Updated by Arthur Kreymer over 4 years ago

Date: Fri, 13 Nov 2015 14:34:40 -0600
From: William F Badgett Jr <>

On what machines is "/web/sites" mounted?
----------------------
Just our SLF6 systems,
minos60-63, minos-data, minos-nearline, minos-slf6

Too ambitious to write to it directlry right now, I think,
Setting up an rsync from the old nusoft for now.

#3 Updated by Arthur Kreymer over 4 years ago

Date: Fri, 13 Nov 2015 15:14:07 -0600
From: William F Badgett Jr <>

What will be the new write mechanism to the /web/sites directories?

Looks like everything is owned by "nobody:nobody" and not even readable
-------------
Date: Fri, 13 Nov 2015 15:19:09 -0600
From: Marc W Mengel <>

http://cd-docdb.fnal.gov/cgi-bin/RetrieveFile?docid=5375&filename=CWS%20Content%20Access%20Instructions.pdf&version=1
-------------------
For now, I plan to set up an rsync from the old area to the new one,
initially using my kerberos ticket, then soon a project keytab .
Most of our process can continue writing to the old areas
without modification in short term.

#4 Updated by Arthur Kreymer over 4 years ago

The certificate seems to work for access to minos.fnal.gov .

KEYTAB=/opt/mindata/kt
export KRB5CCNAME=FILE:/tmp/krb5cc_minos_nusoftweb
KINIT=/usr/krb5/bin/kinit
${KINIT} -5 -A -k -t ${KEYTAB} ${PRINCIPAL}

I will start setting up an rsync.
First want to trim the content a bit,
mainly pre-shutdown controlroom/daqlogs/FAR/om/rootfiles ( 100 GB )

#5 Updated by Arthur Kreymer over 4 years ago

    Preliminary scan of nusoft content, 
    finding large things that are not needed on the web.

     OVERALL

cd /nusoft/app/web/htdoc/minos
du -sm *  # omitting smaller areas

 10052 beammonspill
 57286 calibration
135159 controlroom
 44310 OnlineMonitoring
  9863 OnlineMonitoringFD

     OM

du -sm  OnlineMonitoring/OnlineMonitoringPlots/*
44292    OnlineMonitoring/OnlineMonitoringPlots/plots

2669 directories in plots, these have png images.

    CALIBRATION

48313    FdLinearityFitWorkspace

du -sm /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/*
...
246    /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/fdfit-2008-06
315    /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/fdfit-2008-07

du -sm /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/*
40    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-08
42    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-09
44    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-11

    DAQLOGS

du -sm NEAR/*
3    NEAR/daqMessage.txt
69    NEAR/dbu
1163    NEAR/dds
2449    NEAR/msglog
17771    NEAR/om
757    NEAR/timing
0    NEAR/TouchMe

1    NEAR/om/postscript
17734    NEAR/om/rootfiles
37    NEAR/om/summaries

 9009 root files typically 4 MB, Since Aug 2012

du -sm FAR/*
346    FAR/dbu
170    FAR/dds
2926    FAR/msglog
106158    FAR/om
1    FAR/test.html
3147    FAR/timing
0    FAR/TouchMe

  6880 root files up to 25 MB  Since Aug 2012

#6 Updated by Arthur Kreymer over 4 years ago

I copied the large rootfiles directories from the web tree to the general data area,
creating a symlink for compatibility.
This will free up about 120 GB of web space.

Directories were
NUN=/nusoft/app/web/htdoc/minos/controlroom/daqlogs/NEAR/om/rootfiles
MDN=/minos/data/controlroom/daqlogs/NEAR/om/rootfiles

NUF=/nusoft/app/web/htdoc/minos/controlroom/daqlogs/FAR/om/rootfiles
MDF=/minos/data/controlroom/daqlogs/FAR/om/rootfiles

Copies were done around 14:05 for ND and 15:26 for FD.

The original rootfiles directories were renamed rootfiles-REMOVED.

I will leave these in place for a fallback for a day,
then remove them and copy /nusoft/app/web/htdoc/minos
to /web/sites/minos.fnal.gov/nusoft
for testing.

#7 Updated by Arthur Kreymer over 4 years ago

Here is a summary of advice received last week from Andy Romero.

Access to /web/sites/* is via a special kernel kerberos ticket
that behaves differently than the customary kerberos ticket file.
Perhaps because the kernel is accessing the files, not the user processes.

There is a single kernel structure containing the current kerberos ticket
for each account.
In a shared account we use a project principal to access the web areas.
An interactive login with a forwarded user ticket can override this in the kernel,
possibily removing access to the web area.

The recommended solution is to do a few things :
  • Use an account/host not likely to be used for other work
    • At present, we are using minos@minos60 for controlroom updates
  • Do not forward user tickets when logging into that account
  • Set KRB5CCNAME in the .bash_profile for tha account being used
    • for minos60 , probably should do
      HOST=`hostname -s`
      [ "${HOST}" == 'minosgpvm04' ] && export KRB5CCNAME=FILE:/tmp/krb5cc_minos_web
      * set up a cron job to keep this ticket up to date several times a day
      
    • make a kinweb script in ~/bin
      #!/bin/sh
      export KRB5CCNAME=FILE:/tmp/krb5cc_minos_web
      KEYTAB=/opt/minos/minos-om.keytab
      PRINCIPAL=minos-wh-cr/minos/minos-om.fnal.gov@FNAL.GOV
      KINIT=/usr/krb5/bin/kinit
      ${KINIT} -5 -A  -k -t ${KEYTAB} ${PRINCIPAL}
      
    • run this via cron
      01 02,10,18 * * * ${HOME}/bin/kinweb
      

      I will to ask ECF/SSI to create /opt/minos on all servers

#8 Updated by Arthur Kreymer over 4 years ago

Here's a summary of scans of various areas in nusoft/app/web/htdoc/minos

DATA > du -sm *
    153 acclog
     54 admin
     37 beamdata
  10052 beammonspill
     19 bfield
      8 bluwatch
   1787 cal
  57651 calibration
     56 condor
  19800 controlroom
   2356 database
    233 DataQuality
   2295 dcache
    12 farmgsum
  98499 FDomroot
      1 images
    144 jira
    635 maint
      1 mcas
      1 MinosBatch_AtAGlance
  17921 NDomroot
    202 OldMinosLEDataQualityWebsite
  45761 OnlineMonitoring
  10331 OnlineMonitoringFD
    946 OnlineMonitoring.old
     66 predator
      1 prodmon
    708 validation
      1 vault
    596 WebDocs

Searched for recent changes 11/24, found mainly

acclog
calibration/DriftInfo
calibration/NdLinearityFitWorkspace
calibration/FdLinearityFitWorkspace
database/topdb
dcache/poolsum
validation 
DataQuality
predator
beamdata
OnlineMonitoring

#9 Updated by Arthur Kreymer over 4 years ago

  • Subject changed from move /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov to amove /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov

#10 Updated by Arthur Kreymer over 4 years ago

I will add an executive summary of advice given so far on nusoft web migration
to https://cdcvs.fnal.gov/redmine/projects/admin/wiki/MNS-NUSOFT-WEB

Added rui to the watch list of this Issue.

#11 Updated by Arthur Kreymer over 4 years ago

I have started updating documentation at
https://cdcvs.fnal.gov/redmine/projects/admin/wiki/MNS-NUSOFT-WEB

Will continue after the holidays.

#12 Updated by Arthur Kreymer 15 days ago

  • Estimated time set to 40.00 h
  • Status changed from New to Assigned
  • Subject changed from amove /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov to move /nusoft/app/htdoc/minos to /web/sites/minos.fnal.gov

I am starting to work on this, after a long period of neglect.

I retired in late 2016, but will help with this project,
as it is related to data preservation.

There is an immediate need to free space on the /nusoft/app server.

#13 Updated by Arthur Kreymer 15 days ago

There have been major changes since late 2015 when this project went idle :

The /nusoft/app area remained online after the 2015 shutdown of Sun servers.
The area was moved to new Linux servers, removing the immediate need to migrate.

We established a new /minos/data/web area, served at https://minos.fnal.gov/data
to which the /nusoft/app/web/htdoc/minos files can move.

#14 Updated by Arthur Kreymer 15 days ago

  • Status changed from Assigned to Work in progress

I verified that all the files in /nusoft/app/web/htdoc/minos/MOVED
are present in /minos/data, and removed those directories.
This freed up about 100 GB of space

Details :


NUN=/nusoft/app/web/htdoc/minos/controlroom/daqlogs/NEAR/om/rootfiles
MDN=/minos/data/controlroom/daqlogs/NEAR/om/rootfiles

NUF=/nusoft/app/web/htdoc/minos/controlroom/daqlogs/FAR/om/rootfiles
MDF=/minos/data/controlroom/daqlogs/FAR/om/rootfiles

DATA > du -sm /nusoft/app/web/htdoc/minos/MOVED/*
98598    /nusoft/app/web/htdoc/minos/MOVED/omroot-FARDET
18063    /nusoft/app/web/htdoc/minos/MOVED/omroot-NEARDET

DATA > du -sm  $MDF $MDN
112456    /minos/data/controlroom/daqlogs/FAR/om/rootfiles
39814    /minos/data/controlroom/daqlogs/NEAR/om/rootfiles

ls /nusoft/app/web/htdoc/minos/MOVED/omroot-FARDET | wc -l
6907
ls ${MDF} | wc -l
7771

ls /nusoft/app/web/htdoc/minos/MOVED/omroot-NEARDET | wc -l
9089
ls ${MDN} | wc -l
18377

Many files were written to the rootfiles areas after the originals moved.

cd /nusoft/app/web/htdoc/minos/MOVED

diff -r omroot-NEARDET ${MDN} | tee /dev/shm/Ndiff

wc -l /dev/shm/Ndiff
9289 /dev/shm/Ndiff

head -5 /dev/shm/Ndiff 
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: .htaccess
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_62921_phys.root
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_63294_phys.root
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_63295_phys.root
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_63296_phys.root

grep -v 'Only in /minos/data' /dev/shm/Ndiff

diff -r omroot-FARDET ${MDF} | tee /dev/shm/Fdiff

wc -l /dev/shm/Fdiff
864 /dev/shm/Fdiff

head -5 /dev/shm/Fdiff 
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63617_phys.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63618_cinj.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63619_ped.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63620_phys.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63621_cinj.root

grep -v 'Only in /minos/data' /dev/shm/Fdiff

df -h .
Filesystem            Size  Used Avail Use% Mounted on
if-nas-0.fnal.gov:/nusoft/app
                      1.8T  1.6T  243G  87% /nusoft/app

rm -r omroot-FARDET
rm -r omroot-NEARDET
date
Thu Mar 19 13:17:59 CDT 2020

df -h .
Filesystem            Size  Used Avail Use% Mounted on
if-nas-0.fnal.gov:/nusoft/app
                      1.8T  1.5T  357G  81% /nusoft/app

#15 Updated by Arthur Kreymer 11 days ago

Here is an assessment of recently modified nusoft minos files.

time find /nusoft/app/web/htdoc/minos -type f -mtime -100
/nusoft/app/web/htdoc/minos/calibration/GainReports/FD/1083/.bad_aggregates.png.yrZlPE
/nusoft/app/web/htdoc/minos/calibration/GainReports/FD/560/.gainreports.root.8PQFNs
/nusoft/app/web/htdoc/minos/calibration/GainReports/FD/1317/.bad_aggregates.png.Oed23o
/nusoft/app/web/htdoc/minos/farmgsum/FGS
/nusoft/app/web/htdoc/minos/farmgsum/2019/sum.2019121400
/nusoft/app/web/htdoc/minos/farmgsum/2019/sum.2019121500
. . .
/nusoft/app/web/htdoc/minos/farmgsum/2020/sum.2020032200

real 74m7.750s
user 0m6.213s
sys 1m28.301s

The non-farmgsum files are empty files with future dates :

FILES='
1083/.bad_aggregates.png.yrZlPE
560/.gainreports.root.8PQFNs
1317/.bad_aggregates.png.Oed23o
'
cd /nusoft/app/web/htdoc/minos/calibration/GainReports/FD
ls -l ${FILES} ;}

-rwxrwxrwx 1 mindata e875 0 Jul 20 2084 1083/.bad_aggregates.png.yrZlPE
-rwxrwxrwx 1 jtodd2 e875 0 Nov 22 2062 1317/.bad_aggregates.png.Oed23o
-rwxrwxrwx 1 mindata e875 0 Jul 15 2084 560/.gainreports.root.8PQFNs

I have removed these damaged files.

I will move the farmgsum process to log to /minos/data/web/computing/dh/minfarm

#16 Updated by Arthur Kreymer 11 days ago

The farmgsum files are written by this mindata crontab entry on minos-data.fnal.gov 

crontab -l | grep farmg
15 00  * * *  /grid/fermiapp/minos/scripts/farmgsum_log quiet

cp -a farmgsum_log farmgsum_log.20150101

changeed farmgsum_log LOG area from
LOG=/nusoft/app/web/htdoc/minos/farmgsum

to
/minos/data/web/computing/dh/farmgsum

du -sm /nusoft/app/web/htdoc/minos/farmgsum
136    /nusoft/app/web/htdoc/minos/farmgsum

    COPIED OLD FILES

date
cp -a /nusoft/app/web/htdoc/minos/farmgsum /minos/data/web/computing/dh/farmgsum

Mon Mar 23 15:17:08 CDT 2020

diff -r  /nusoft/app/web/htdoc/minos/farmgsum /minos/data/web/computing/dh/farmgsum

du -sm /minos/data/web/computing/dh/farmgsum
136    /minos/data/web/computing/dh/farmgsum

date
/grid/fermiapp/minos/scripts/farmgsum_log

    Summarizing /minos/data/minfarm/*cat Mon Mar 23 15:19:38 CDT 2020  
...

https://minos.fnal.gov/data/computing/dh/farmgsum/2020/sum.2020032315

I will wait for the cron job to run tonight, then remove
/nusoft/app/web/htdoc/minos/farmgsum/nusoft/app/web/htdoc/minos/farmgsum

#17 Updated by Arthur Kreymer 10 days ago

    removed nusoft farmgsum files

mindata@minos-data.fnal.gov

diff -r  /nusoft/app/web/htdoc/minos/farmgsum /minos/data/web/computing/dh/farmgsum

Only in /minos/data/web/computing/dh/farmgsum/2020: sum.2020032315
Only in /minos/data/web/computing/dh/farmgsum/2020: sum.2020032400
diff -r /nusoft/app/web/htdoc/minos/farmgsum/FGS /minos/data/web/computing/dh/farmgsum/FGS
2c2
<     Summarizing /minos/data/minfarm/*cat Mon Mar 23 00:15:01 CDT 2020  
---
>     Summarizing /minos/data/minfarm/*cat Tue Mar 24 00:15:02 CDT 2020  

date
rm  -r  /nusoft/app/web/htdoc/minos/farmgsum

Tue Mar 24 23:56:33 CDT 2020



Also available in: Atom PDF