Support #10919

move /nusoft/app/htdoc/minos to /web/sites/

Added by Arthur Kreymer over 5 years ago. Updated 6 days ago.

Start date:
Due date:
% Done:


Estimated time:
40.00 h
Spent time:


We need to move /nusoft/app/htdoc/minos to /web/sites/
This is due by Nov 20, this year.
This is part of the shutdown of the old Sun web servers.


Support #10921: ND DCS, DAQ, and Timing web pages all run on nusoft spaceNew


#1 Updated by Arthur Kreymer over 5 years ago

What I think we might do right now is to rsync
to /web/sites/

This way all present scripts can continue to push content,
we can update them later.

We first need to trim down controlroom/daqlogs

du -sm /nusoft/app/web/htdoc/minos/controlroom/daqlogs/*
112674 /nusoft/app/web/htdoc/minos/controlroom/daqlogs/FAR
22141 /nusoft/app/web/htdoc/minos/controlroom/daqlogs/NEAR

I have already made a static copy of /nusoft/app/web/cgi-bin/minos
to /web/sites/

fluffy.cgi and status work,
the other seem to depend on paths that will shift.

#2 Updated by Arthur Kreymer over 5 years ago

Date: Fri, 13 Nov 2015 14:34:40 -0600
From: William F Badgett Jr <>

On what machines is "/web/sites" mounted?
Just our SLF6 systems,
minos60-63, minos-data, minos-nearline, minos-slf6

Too ambitious to write to it directlry right now, I think,
Setting up an rsync from the old nusoft for now.

#3 Updated by Arthur Kreymer over 5 years ago

Date: Fri, 13 Nov 2015 15:14:07 -0600
From: William F Badgett Jr <>

What will be the new write mechanism to the /web/sites directories?

Looks like everything is owned by "nobody:nobody" and not even readable
Date: Fri, 13 Nov 2015 15:19:09 -0600
From: Marc W Mengel <>
For now, I plan to set up an rsync from the old area to the new one,
initially using my kerberos ticket, then soon a project keytab .
Most of our process can continue writing to the old areas
without modification in short term.

#4 Updated by Arthur Kreymer over 5 years ago

The certificate seems to work for access to .

export KRB5CCNAME=FILE:/tmp/krb5cc_minos_nusoftweb
${KINIT} -5 -A -k -t ${KEYTAB} ${PRINCIPAL}

I will start setting up an rsync.
First want to trim the content a bit,
mainly pre-shutdown controlroom/daqlogs/FAR/om/rootfiles ( 100 GB )

#5 Updated by Arthur Kreymer over 5 years ago

    Preliminary scan of nusoft content, 
    finding large things that are not needed on the web.


cd /nusoft/app/web/htdoc/minos
du -sm *  # omitting smaller areas

 10052 beammonspill
 57286 calibration
135159 controlroom
 44310 OnlineMonitoring
  9863 OnlineMonitoringFD


du -sm  OnlineMonitoring/OnlineMonitoringPlots/*
44292    OnlineMonitoring/OnlineMonitoringPlots/plots

2669 directories in plots, these have png images.


48313    FdLinearityFitWorkspace

du -sm /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/*
246    /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/fdfit-2008-06
315    /nusoft/app/web/htdoc/minos/calibration/FdLinearityFitWorkspace/fdfit-2008-07

du -sm /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/*
40    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-08
42    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-09
44    /nusoft/app/web/htdoc/minos/calibration/NdLinearityFitWorkspace/ndfit-2007-11


du -sm NEAR/*
3    NEAR/daqMessage.txt
69    NEAR/dbu
1163    NEAR/dds
2449    NEAR/msglog
17771    NEAR/om
757    NEAR/timing
0    NEAR/TouchMe

1    NEAR/om/postscript
17734    NEAR/om/rootfiles
37    NEAR/om/summaries

 9009 root files typically 4 MB, Since Aug 2012

du -sm FAR/*
346    FAR/dbu
170    FAR/dds
2926    FAR/msglog
106158    FAR/om
1    FAR/test.html
3147    FAR/timing
0    FAR/TouchMe

  6880 root files up to 25 MB  Since Aug 2012

#6 Updated by Arthur Kreymer over 5 years ago

I copied the large rootfiles directories from the web tree to the general data area,
creating a symlink for compatibility.
This will free up about 120 GB of web space.

Directories were


Copies were done around 14:05 for ND and 15:26 for FD.

The original rootfiles directories were renamed rootfiles-REMOVED.

I will leave these in place for a fallback for a day,
then remove them and copy /nusoft/app/web/htdoc/minos
to /web/sites/
for testing.

#7 Updated by Arthur Kreymer over 5 years ago

Here is a summary of advice received last week from Andy Romero.

Access to /web/sites/* is via a special kernel kerberos ticket
that behaves differently than the customary kerberos ticket file.
Perhaps because the kernel is accessing the files, not the user processes.

There is a single kernel structure containing the current kerberos ticket
for each account.
In a shared account we use a project principal to access the web areas.
An interactive login with a forwarded user ticket can override this in the kernel,
possibily removing access to the web area.

The recommended solution is to do a few things :
  • Use an account/host not likely to be used for other work
    • At present, we are using minos@minos60 for controlroom updates
  • Do not forward user tickets when logging into that account
  • Set KRB5CCNAME in the .bash_profile for tha account being used
    • for minos60 , probably should do
      HOST=`hostname -s`
      [ "${HOST}" == 'minosgpvm04' ] && export KRB5CCNAME=FILE:/tmp/krb5cc_minos_web
      * set up a cron job to keep this ticket up to date several times a day
    • make a kinweb script in ~/bin
      export KRB5CCNAME=FILE:/tmp/krb5cc_minos_web
      ${KINIT} -5 -A  -k -t ${KEYTAB} ${PRINCIPAL}
    • run this via cron
      01 02,10,18 * * * ${HOME}/bin/kinweb

      I will to ask ECF/SSI to create /opt/minos on all servers

#8 Updated by Arthur Kreymer over 5 years ago

Here's a summary of scans of various areas in nusoft/app/web/htdoc/minos

DATA > du -sm *
    153 acclog
     54 admin
     37 beamdata
  10052 beammonspill
     19 bfield
      8 bluwatch
   1787 cal
  57651 calibration
     56 condor
  19800 controlroom
   2356 database
    233 DataQuality
   2295 dcache
    12 farmgsum
  98499 FDomroot
      1 images
    144 jira
    635 maint
      1 mcas
      1 MinosBatch_AtAGlance
  17921 NDomroot
    202 OldMinosLEDataQualityWebsite
  45761 OnlineMonitoring
  10331 OnlineMonitoringFD
    946 OnlineMonitoring.old
     66 predator
      1 prodmon
    708 validation
      1 vault
    596 WebDocs

Searched for recent changes 11/24, found mainly


#9 Updated by Arthur Kreymer over 5 years ago

  • Subject changed from move /nusoft/app/htdoc/minos to /web/sites/ to amove /nusoft/app/htdoc/minos to /web/sites/

#10 Updated by Arthur Kreymer over 5 years ago

I will add an executive summary of advice given so far on nusoft web migration

Added rui to the watch list of this Issue.

#11 Updated by Arthur Kreymer over 5 years ago

I have started updating documentation at

Will continue after the holidays.

#12 Updated by Arthur Kreymer about 1 year ago

  • Estimated time set to 40.00 h
  • Status changed from New to Assigned
  • Subject changed from amove /nusoft/app/htdoc/minos to /web/sites/ to move /nusoft/app/htdoc/minos to /web/sites/

I am starting to work on this, after a long period of neglect.

I retired in late 2016, but will help with this project,
as it is related to data preservation.

There is an immediate need to free space on the /nusoft/app server.

#13 Updated by Arthur Kreymer about 1 year ago

There have been major changes since late 2015 when this project went idle :

The /nusoft/app area remained online after the 2015 shutdown of Sun servers.
The area was moved to new Linux servers, removing the immediate need to migrate.

We established a new /minos/data/web area, served at
to which the /nusoft/app/web/htdoc/minos files can move.

#14 Updated by Arthur Kreymer about 1 year ago

  • Status changed from Assigned to Work in progress

I verified that all the files in /nusoft/app/web/htdoc/minos/MOVED
are present in /minos/data, and removed those directories.
This freed up about 100 GB of space

Details :



DATA > du -sm /nusoft/app/web/htdoc/minos/MOVED/*
98598    /nusoft/app/web/htdoc/minos/MOVED/omroot-FARDET
18063    /nusoft/app/web/htdoc/minos/MOVED/omroot-NEARDET

DATA > du -sm  $MDF $MDN
112456    /minos/data/controlroom/daqlogs/FAR/om/rootfiles
39814    /minos/data/controlroom/daqlogs/NEAR/om/rootfiles

ls /nusoft/app/web/htdoc/minos/MOVED/omroot-FARDET | wc -l
ls ${MDF} | wc -l

ls /nusoft/app/web/htdoc/minos/MOVED/omroot-NEARDET | wc -l
ls ${MDN} | wc -l

Many files were written to the rootfiles areas after the originals moved.

cd /nusoft/app/web/htdoc/minos/MOVED

diff -r omroot-NEARDET ${MDN} | tee /dev/shm/Ndiff

wc -l /dev/shm/Ndiff
9289 /dev/shm/Ndiff

head -5 /dev/shm/Ndiff 
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: .htaccess
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_62921_phys.root
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_63294_phys.root
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_63295_phys.root
Only in /minos/data/controlroom/daqlogs/NEAR/om/rootfiles: mon_rootfile_63296_phys.root

grep -v 'Only in /minos/data' /dev/shm/Ndiff

diff -r omroot-FARDET ${MDF} | tee /dev/shm/Fdiff

wc -l /dev/shm/Fdiff
864 /dev/shm/Fdiff

head -5 /dev/shm/Fdiff 
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63617_phys.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63618_cinj.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63619_ped.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63620_phys.root
Only in /minos/data/controlroom/daqlogs/FAR/om/rootfiles: mon_rootfile_63621_cinj.root

grep -v 'Only in /minos/data' /dev/shm/Fdiff

df -h .
Filesystem            Size  Used Avail Use% Mounted on
                      1.8T  1.6T  243G  87% /nusoft/app

rm -r omroot-FARDET
rm -r omroot-NEARDET
Thu Mar 19 13:17:59 CDT 2020

df -h .
Filesystem            Size  Used Avail Use% Mounted on
                      1.8T  1.5T  357G  81% /nusoft/app


df -h .
Filesystem                Size  Used Avail Use% Mounted on  2.9T  2.3T  573G  81% /nashome

#15 Updated by Arthur Kreymer about 1 year ago

Here is an assessment of recently modified nusoft minos files.

time find /nusoft/app/web/htdoc/minos -type f -mtime -100
. . .

real 74m7.750s
user 0m6.213s
sys 1m28.301s

The non-farmgsum files are empty files with future dates :

cd /nusoft/app/web/htdoc/minos/calibration/GainReports/FD
ls -l ${FILES} ;}

-rwxrwxrwx 1 mindata e875 0 Jul 20 2084 1083/.bad_aggregates.png.yrZlPE
-rwxrwxrwx 1 jtodd2 e875 0 Nov 22 2062 1317/.bad_aggregates.png.Oed23o
-rwxrwxrwx 1 mindata e875 0 Jul 15 2084 560/.gainreports.root.8PQFNs

I have removed these damaged files.

I will move the farmgsum process to log to /minos/data/web/computing/dh/minfarm

#16 Updated by Arthur Kreymer about 1 year ago

The farmgsum files are written by this mindata crontab entry on 

crontab -l | grep farmg
15 00  * * *  /grid/fermiapp/minos/scripts/farmgsum_log quiet

cp -a farmgsum_log farmgsum_log.20150101

changeed farmgsum_log LOG area from


du -sm /nusoft/app/web/htdoc/minos/farmgsum
136    /nusoft/app/web/htdoc/minos/farmgsum


cp -a /nusoft/app/web/htdoc/minos/farmgsum /minos/data/web/computing/dh/farmgsum

Mon Mar 23 15:17:08 CDT 2020

diff -r  /nusoft/app/web/htdoc/minos/farmgsum /minos/data/web/computing/dh/farmgsum

du -sm /minos/data/web/computing/dh/farmgsum
136    /minos/data/web/computing/dh/farmgsum


    Summarizing /minos/data/minfarm/*cat Mon Mar 23 15:19:38 CDT 2020  

I will wait for the cron job to run tonight, then remove

#17 Updated by Arthur Kreymer about 1 year ago

    removed nusoft farmgsum files

diff -r  /nusoft/app/web/htdoc/minos/farmgsum /minos/data/web/computing/dh/farmgsum

Only in /minos/data/web/computing/dh/farmgsum/2020: sum.2020032315
Only in /minos/data/web/computing/dh/farmgsum/2020: sum.2020032400
diff -r /nusoft/app/web/htdoc/minos/farmgsum/FGS /minos/data/web/computing/dh/farmgsum/FGS
<     Summarizing /minos/data/minfarm/*cat Mon Mar 23 00:15:01 CDT 2020  
>     Summarizing /minos/data/minfarm/*cat Tue Mar 24 00:15:02 CDT 2020  

rm  -r  /nusoft/app/web/htdoc/minos/farmgsum

Tue Mar 24 23:56:33 CDT 2020

#18 Updated by Arthur Kreymer about 1 year ago

I updated the plan at
Directory sizes had been sadly out of date, vintage 2015.

I removed the tiny admin directory, no longer needed.

I moved the largest area, OnlineMonitoring, over 100 GB.
Renamed the original under MOVED, will remove MOVED/* after about a week.

The calibration area will move last.
It has thousands of html files with explicit links to nusoft.
Calibrations scripts should be updated to use relative links.

#19 Updated by Arthur Kreymer about 1 year ago

All but two directories have been copied to /minos/data/web,
and moved into the /nusoft/app/web/htdoc/minos/MOVED pending deletion.

Two areas remain :
calibration - need to review file ownership
controlroom - need to review existing partial move

#20 Updated by Arthur Kreymer about 1 year ago

controlroom was moved today.
Ownership of the files was minos, and is now mindata.

There was already an active controlroom/webdcs area.
This was retained,he moved.
The copied webdcs area was renamed webdcs-2017

The last remaining area to move is calibration.
There are over 1000 calibration html files with explicit links to nusoft.
These links will need to be corrected after the copies are made.
Calibration scripts may need to be updated.

#21 Updated by Arthur Kreymer about 1 year ago

I created a stash of all original calibration *.html and *.htm files
The copy is in /minos/data/web/nusoft/html/calibration.
I am making fair progress removing http://nusoft references in the copied files.

I removed the minos account crontab on minos-nearline.
This had been running TimeGoblinLooter/loot on /nusoft daq logs.

#22 Updated by Arthur Kreymer about 1 year ago

I used script ${NLO}/fix-gain to update the bulk of remaining full URL in calibration/GainReports.
I edited the ND and FD top level index.html files by hand.

I updated the calibration web page to point to /minos/data/web.

I copied a few stray top level ${NUM} files to ${MDW}/maint.

All /nusoft/app/web/htdoc/minos files are now copied to /minos/data/web,
All /nusoft/app/web/htdoc/minos files have been moved down into MOVED/.

It no issues arise in a few weeks, we should remove the MOVED directory,
releasing the minos space in /nusoft/app.

#23 Updated by Arthur Kreymer about 1 year ago

I removed the individual database symlinks in /nusoft/app/web/htdoc/minos
They are not used by the web server, and serve no purpose.

I added an overall database link to /minos/data/web/database,
to match the other moved directories.

I added an informational page at the top of nusoft/minos, named

#24 Updated by Arthur Kreymer 8 months ago

I was about to remove the MOVED files owned by mindata.

I checked for recent access, doing in MOVED

DIRS=$(find . maxdepth 1 -type d -user mindata | cut -c 3 | sort )

latesta() { find ${1} -type f -print0 | xargs -0 stat --format '%X :%x %n' \ | sort nr | cut -d: -f2 | head -3 ; }

for DIR in ${DIRS} ; do echo ; latesta ${DIR} ; done

I found a lot of currently accessed files.

This is probably because the files are leaking to we web, probably getting crawled.
I left the access open when migrating.
Time to cut it off.

Created MOVED/.htaccess with
Deny to All

This seems to work.

Will let this stew for another week, then removed the MOVED files owned by mindata.

#25 Updated by Arthur Kreymer 8 months ago

To evade web crawlers with a memory of the old webMOVED path
I renamed the /nuspft/app/web/htdoc/minos/MOVED directory to REMOVE.

date ; mv MOVED REMOVE
Wed Sep 9 16:14:05 CDT 2020

#26 Updated by Arthur Kreymer 8 months ago

Checked that there was no access to /nusoft/app/web/htdoc/minos/REMOVE
after this area was renamed away from web crawlers 09/ 09

cd  /nusoft/app/web/htdoc/minos/REMOVE

latesta() { find ${1} -type f -atime -7 -print0 | xargs -0 stat --format '%X :%x %n' \
| sort -nr | cut -d: -f2- | head  ; }

date ; time latesta .

Mon Sep 14 13:41:35 CDT 2020

2020-09-13 19:53:56.083318657 -0500 ./.htaccess
2020-09-09 15:01:36.461539942 -0500 ./calibration/.htaccess
2020-09-09 14:40:16.906028775 -0500 ./jira/.htaccess
2020-09-08 18:54:36.536084254 -0500 ./WebDocs/HEAD/doxygen/loon/html/classMiniMaker-members.html
2020-09-08 18:51:55.015206715 -0500 ./DataQuality/xstyles/xprint.css
2020-09-08 18:51:54.980879245 -0500 ./DataQuality/xstyles/ximages/diamond-gray.gif
2020-09-08 18:51:54.934470887 -0500 ./DataQuality/xstyles/xstyle.css
2020-09-08 18:51:54.882108856 -0500 ./DataQuality/index.html
2020-09-08 18:51:12.927303804 -0500 ./WebDocs/HEAD/doxygen/loon/html/classEVD.html
2020-09-08 18:48:38.307681913 -0500 ./auth/.htpasswd

real    110m35.315s
user    0m11.619s
sys    2m19.477s

We can remove content from REMOVE directories.

#27 Updated by Arthur Kreymer 8 months ago

There continues to be no recent read accesses to nusoft files but the top .htaccess

I have removed the following areas owned by mindata :

acclog auth beamdata beammonspill bfield bluwatch cal condor database DataQuality dcache maint mcas OldMinosLEDataQualityWebsite OnlineMonitoring OnlineMonitoringFD OnlineMonitoring.old predator prodmon validation

calibration is hard to remove, as most directories are owned by tagg.

There were similar issues with other of these directories, but involving shared accounts,
or user files rather than directories. I was able to work around those issues.

There are a few more directories to deal with, not owned by mindata.


 calibration          mindata 
                      tagg         no    tagg items pending
 controlroom          minos        no
 images               badgett      no
 jira                 minos        no
 MinosBatch_AtAGlance lena         no    lena account not available
 vault                minosraw     no
 WebDocs              minossoft    no

See full table of areas in

#28 Updated by Arthur Kreymer 7 months ago

I removed all but the tagg files from calibration.
I sent email to tagg asking that the remaining files be removed :

Removal of relocated nusoft minos calibration files

We moved the calibration web pages
from /nusoft/app/web/htdoc/minos/calibration
to /minos/data/web/calibration
and moved the files out of the way to /nusoft/app/web/htdoc/minos/REMOVE

I am trying to remove the original nusoft files
but many directories are owned by user tagg.

At your convenience, could you please :

Login to

cd /nusoft/app/web/htdoc/minos/REMOVE/calibration
rm -r NdLinearityFitWorkspace/*
rm -r FdLinearityFitWorkspace/*

This will take a little while, removing about 70 GB in 427768 files.

Thanks !

#29 Updated by Arthur Kreymer 7 months ago

Tagg, Nathaniel via
9:44 AM (4 hours ago)
to , Art, , Robert, , , , , , ,

Hi Art,

Running that now. Thanks for your help.

#30 Updated by Arthur Kreymer 7 months ago

Thanks Nathaniel !

I have removed the REMOVE/calibration area.

We will need administrative help to remove REMOVE/MinosBatch_AtAGlance

#31 Updated by Arthur Kreymer 7 months ago

  • Status changed from Work in progress to Resolved

We got access to images and MinosBatch_AtAGlance with RITM1024266.

I have removed the REMOVE area.

I updated

I updated the 2020/03/19 note above, showing free space in /nusoft/app

There is now 573 GB free, formerly 243.

So Minos have freed up to 330 GB.
This is consistent with the 323395 Mbytes seen in June of 2019.

#32 Updated by Arthur Kreymer 6 days ago

I made a final reference copy of the nusoft minos cgi-bin files,
and removed the directory.


rm -r /nusoft/app/web/cgi-bin/minos ; date
rm: cannot remove \u2018/nusoft/app/web/cgi-bin/minos\u2019: Permission denied
Thu Apr 29 10:17:55 CDT 2021

ls -ld   /nusoft/app/web/cgi-bin
drwxr-xr-x 6 nusoft nusoft 2048 Mar 25  2014 /nusoft/app/web/cgi-bin


rmdir /nusoft/app/web/cgi-bin/minos ; date
Thu Apr 29 11:17:01 CDT 2021

Also available in: Atom PDF