Project

General

Profile

Migration from /nusoft/app/web/htdoc/minos to /web/sites/minos.fnal.gov started in 2015

Migration from /nusoft/app/web/htdoc/minos to /minos/data/web completed in 2020

The /nusoft/app/web/htdoc area was set up as a (successful) prototype
for the current central web servers.
The intended long term customer was the shared neutrino software project,
and not the specific experiments.

Early testing was done in directories dune, lariat, lbne, minerva, minos and nova

As of 2020/03/19 , directory sizes were (MBytes)

     62 dune
    352 lariat
     16 lbne
 304222 minerva
 322535 minos
 473583 nova
1119651 total

Work is being tracked in https://cdcvs.fnal.gov/redmine/issues/10919

Directories are owned by a mix of minos ( DAQ ) , and mindata ( OFFLINE ).

General documentation

Some documentation is still buried in the work details logged in

https://cdcvs.fnal.gov/redmine/issues/10919 (nusoft) and
https://cdcvs.fnal.gov/redmine/issues/10920 (www-numi)

minos.fnal.gov/data

Files in /minos/data/web are served to the web at https://minos.fnal.gov/data
This is where the /nusoft/app/htdoc/minos files are being moved.

OTHER WEB AREAS

www-numi.fnal.gov

www-numi was migrated from AFS to the new servers for use as-is around 2016.
The source files are /web/sites/w/www-numi.fnal.gov .

minos.fnal.gov

The new minos site is https://minos.fnal.gov

The site is hosted by new Linux Apache servers, from /web/sites/minos.fnal.gov .
Someday they may move this to /web/sites/m/minos.fnal.gov
with a compatibility symlink from the present path. This is not scheduled as of 2020.

These files are accessed using an NFS 4.1 mount, requiring a registered active Kerberos ticket.
http://cd-docdb.fnal.gov/cgi-bin/ShowDocument?docid=5375
For interactive editing, get registered and log into any Minos interactive system.
Registry is summarized below.

Automated processes can use a services principal to access this area.
We have presently registered
Access can be tricky, this is a kernel level ticket shared by all of an account's processes on a given host
Logging in interactively with a forwarded ticket can flip the ticket, and there seem to be time delays.

For automatic processes we will be able to also have regular NFS3 mounted areas served,
and mapped to the desired path.
For now we will mount existing areas from /nusoft/app/web/htdoc/minos/...
Later these will likely move to /minos/data/web/...

NUSOFT MINOS directories

stray is the count of files not owned by the directory owner ,scanned 2020-04-05

Directory MB Latest Owner, stray Moved Comment
acclog 259 2019-06-04 mindata 2020-04-06 first use of copy and move scripts
admin 56 2015-11-24 mindata 10 2020-04-03 moved recentdir to NLO/recentdir-2015, removed kreymer
auth 1 2016-01-11 mindata 2020-04-06 web auth files, for reference ?
beamdata 390 2019-07-09 mindata 2020-04-06 3 nearly identical CF related listings from 2019-07-09
beammonspill 46504 2019-07-09 mindata 2020-04-07
bfield 60 2019-08-13 mindata 2020-04-07
bluwatch 25 2011-03-29 mindata 2020-04-07
cal 2131 2012-07-05 mindata 2020-04-07
calibration 75582 2019-08-02 mindata 429902
> mindata
2020-04-09 most files were owned by tagg
changed html to relative links and moved 04/16
condor 78 2015-06-18 mindata 2020-04-07
controlroom 25941 2019-09-30 minos 5
> mindata
2020-04-08 webdcs > webdcs-2017 see also /minos/data/controlroom
database 8004 2016-01-21 minosdb 2020-04-07 copied with copydb, minosdb as mindata
to match existing MDW directories
DataQuality 410 2019-07-09 mindata 2020-04-07
dcache 7120 2017-11-19 mindata 2020-04-07
farmgsum 136 2020-03-23 mindata 2020-03-23 to computing/dh/farmgsum
updated farmgsum script
images 1 2013-07-24 badgett
> mindata
2020-04-07
jira 279 2012-12-03 minos
mindata
2020-04-07 static archive
maint 638 2012-06-18 mindata 2020-04-07
mcas 1 2015-02-05 mindata 2020-04-07
MinosBatch_AtAGlance 2 2014-04-05 lena
> mindata
2020-04-06 njobs.txt dated 2017-08-17. Fixed absolute paths
OldMinosLEDataQualityWebsite 251 2013-07-09 mindata 2020-04-07
OnlineMonitoring 102166 2016-02-03 mindata 2020-04-04 > OnlineMonitoring, some duplication of OM
OnlineMonitoringFD 27247 2016-02-03 mindata 2020-04-07
OnlineMonitoring.old 2008 2014-11-26 mindata 24 2020-04-07
predator 100 2017-01-22 mindata 2020-04-07
prodmon 1 2012-10-01 mindata 2020-04-07
validation 1484 2019-07-09 mindata 3487 2020-04-06 made index.html paths relative
archive directory was owned by 500.500
vault 1 2017-04-05 minosraw 2017-04-10 > vault
WebDocs 1108 2013-08-28 minsoft
> mindata
2020-04-07
TOTAL 299782 TOTAL SPACE

In interactive sessions working on this, I am setting

MDW=/minos/data/web
NUM=/nusoft/app/web/htdoc/minos
NLO=/minos/app/home/mindata/log/nusoft

Find the latest file in a directory using this function

latest() { find ${1} -type f -print0 | xargs -0 stat --format '%Y :%y %n' \
| sort -nr | cut -d: -f2- | head  ; }

FULL LINKS

There are 1410 files in NUM with explicit links to nusoft.fnal.gov .
I plan to copy the files to MDW, then correct the links.
Almost all the effort will be for the calibration files.

The full list of links is ${NLO}/nulinks.log

This was created with

set nohup
{ date
 find . -name *.html -exec grep nusoft {} \; -exec echo {} \; 
 date
}  > ${HOME}/log/nusoft/nulinks.log 2>&1 &

grep ^./ nulinks.log | wc -l
1410

The directory breakdown is


DIRS=`grep ^./ nulinks.log  | cut -f 1-3 -d / | sort -u`

for DIR in ${DIRS} ; do
    ND=`grep ^${DIR} nulinks.log | wc -l`
    printf "%5d ${DIR}\n" ${ND}
done

    3 ./calibration/DriftInfo
   43 ./calibration/FdLinearityFitWorkspace
 1279 ./calibration/GainReports
   77 ./calibration/NdLinearityFitWorkspace
    3 ./calibration/OldDriftInfo
    1 ./MinosBatch_AtAGlance/MinosBatch_atAGlance.html
    2 ./validation/archive
    1 ./validation/far
    1 ./validation/near

I created a reference copy of all calibration *.html and *.htm files
This is in ${MDW}/nusoft/html/calibration.
The copy was made with new script ${NLO}/copy-html

The links were repaired by scripts found under ${NLO} :
fix-fdfit
fix-ndfit
fix-gain

Details of this work are found in /nashome/k/kreymer/LOG
https://home.fnal.gov/~kreymer/act/LOG

OnlineMonitoring

OnlineMonitoring/OnlineMonitoringPlots/plots was first copied
to ${MDW}/OM in January of 2016.
This was the initial area tested with the new server.

The controlroom web links were updated to MDW at that time.

COPY PLAN  
    There are too many differing binary files to directly merge short term.
    So we will just copy the OnlineMonitoring* directories directly to MDW.
    We can consider merging these in to OM and OMfar later.

The original merging plan which was abandoned due to differing binaries :
    Verify that MDW plots is more complete than NUM plots, with diff
    remove the old plots directory
    copy NUM/OnlineMonitoring to MDW/OnlineMonitoring
    Move the more complete MDW/OM plots area into OnlineMonitoring
    remove empty OM tree 
    Rename OnlineMonitoring to OM

Moved as follows :

set nohup
{ date ; time cp -ax ${NUM}/OnlineMonitoring     ${MDW}/OnlineMonitoring ; date ; }
Sat Apr  4 13:39:42 CDT 2020
real    245m56.695s
user    0m50.159s
sys    12m20.616s
Sat Apr  4 17:45:39 CDT 2020

diff -r ${NUM}/OnlineMonitoring     ${MDW}/OnlineMonitoring

mv    ${NUM}/OnlineMonitoring  ${NUM}/MOVED/OnlineMonitoring 
ln -s ${MDW}/OnlineMonitoring  ${NUM}/OnlineMonitoring
Sun Apr  5 22:44:30 CDT 2020

To make this consistent, I created self-logging scripts in ${NLO}
The scripts append to ${NLO}/movelogs/<directory>.log
The scripts show the content of the log when they finish, via less.

Use like this :

cd ${NLO}
./copy <directory>
./move <directory>

MAINT FILES

There were three stray files at the top level :
beammonspill_backup.sh
beammonspill_backup.sh.~1~
nohup.out


I copied these to ${MDW}/maint, and moved them to ${NUM}/MOVED