Project

General

Profile

Support #16006

change vault dd to ftp

Added by Arthur Kreymer over 3 years ago. Updated 2 months ago.

Status:
Closed
Priority:
Low
Start date:
03/28/2017
Due date:
04/14/2017
% Done:

100%

Estimated time:
10.00 h
Duration: 18

Description

Raw data files are copied monthly to /pnfs/minos/sfa/vault
These copies have been partial starting with 2016-06.

o Move the scripts from minosraw@minos-slf6 to minos-data
o Change the copy method to ifdh cp
o Repeal and replace the invalid files

History

#1 Updated by Arthur Kreymer over 3 years ago


    REMOVE THE EMPTY ARCHIVE FILES

find /pnfs/minos/sfa/vault/neardet_data/2016* -size 0 -exec ls -l {} \;

-rw-r--r-- 1 minosraw e875 0 Jul  3  2016 /pnfs/minos/sfa/vault/neardet_data/2016-06/N00064974_0009.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Aug  3  2016 /pnfs/minos/sfa/vault/neardet_data/2016-07/N00070017_0004.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Sep  3  2016 /pnfs/minos/sfa/vault/neardet_data/2016-08/N00070249_0012.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Oct  3 20:12 /pnfs/minos/sfa/vault/neardet_data/2016-09/N00070427_0014.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Nov  3 20:11 /pnfs/minos/sfa/vault/neardet_data/2016-10/N00070631_0000.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Dec  3 20:12 /pnfs/minos/sfa/vault/neardet_data/2016-11/N00070940_0002.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Jan  3 20:12 /pnfs/minos/sfa/vault/neardet_data/2016-12/N00062920_0015.mdaq.root

find /pnfs/minos/sfa/vault/neardet_data/2016* -size 0 -exec echo "rm -I {}" \;

date
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-06/N00064974_0009.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-07/N00070017_0004.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-08/N00070249_0012.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-09/N00070427_0014.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-10/N00070631_0000.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-11/N00070940_0002.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-12/N00062920_0015.mdaq.root
date

cut/pasted these

Sat Apr  1 09:45:31 CDT 2017
Sat Apr  1 09:45:31 CDT 2017

find /pnfs/minos/sfa/vault/neardet_data/2017* -size 0 -exec ls -l {} \;
-rw-r--r-- 1 minosraw e875 0 Mar  3 21:01 /pnfs/minos/sfa/vault/neardet_data/2017-02/N00071670_0012.mdaq.root

find /pnfs/minos/sfa/vault/neardet_data/2017* -size 0 -exec echo "rm -I {}" \;

rm -I /pnfs/minos/sfa/vault/neardet_data/2017-02/N00071670_0012.mdaq.root
date
Sat Apr  1 09:47:38 CDT 2017

   Removed the partial file found in the previous scan

ls -l /pnfs/minos/neardet_data/2017-01/N00071395_0005.mdaq.root \
      /pnfs/minos/sfa/vault/neardet_data/2017-01/N00071395_0005.mdaq.root 

-rw-rw-r-- 1 minosraw e875 58803266 Jan  1 07:37 /pnfs/minos/neardet_data/2017-01/N00071395_0005.mdaq.root
-rw-r--r-- 1 minosraw e875 10551296 Feb  3 20:11 /pnfs/minos/sfa/vault/neardet_data/2017-01/N00071395_0005.mdaq.roo

rm -I /pnfs/minos/sfa/vault/neardet_data/2017-01/N00071395_0005.mdaq.root
date
Sat Apr  1 09:51:48 CDT 2017

    REMOVE THE 2014-05 corrupted file

ls -l /pnfs/minos/neardet_data/2014-05/N00058872_0006.mdaq.root \
      /pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root 

-rw-r--r-- 1 minosraw e875 133454372 May 17  2014 /pnfs/minos/neardet_data/2014-05/N00058872_0006.mdaq.root
-rw-r--r-- 1 minosraw e875 133454372 Nov 10  2014 /pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.ro

RAW > . ./setups.sh
RAW > setup encp -q stken

BAS=/pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root
BAD=/pnfs/minos/BAD/neardet_data/sfa_N00058872_0006.mdaq.root

date ; enmv ${BAS} ${BAD}
Sat Apr  1 10:00:51 CDT 2017
ERROR: OS ERROR Unable to access file /pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root: [Errno 13] Permission denied: '/pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root'

   shift it in place ?

BAS=/pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root
BAD=/pnfs/minos/sfa/vault/neardet_data/2014-05/BAD_N00058872_0006.mdaq.root

date ; enmv ${BAS} ${BAD}
Sat Apr  1 10:03:50 CDT 2017

    TRY IN sfa/vault

mkdir /pnfs/minos/sfa/vault/BAD

BAS=/pnfs/minos/sfa/vault/neardet_data/2014-05/BAD_N00058872_0006.mdaq.root
BAD=/pnfs/minos/sfa/vault/BAD/N00058872_0006.mdaq.root

date ; enmv ${BAS} ${BAD}
Sat Apr  1 10:10:52 CDT 2017

#2 Updated by Arthur Kreymer over 3 years ago

Will leave this activity in the minosraw account to allow direct mkdir.
Got /opt/minosraw directories for keytabs created with RITM0546594 04/03 .

scp mindata@minos-nearline:/opt/mindata/kt /opt/minosraw/daqdcp-nd.keytab

#3 Updated by Arthur Kreymer over 3 years ago

Scan of missing vault files, done on March 26

for MONTH in ${MONTHS} ; do
    echo ${MONTH}

RAW=/pnfs/minos/${DET}/${MONTH}
SFA=/pnfs/minos/sfa/vault/${DET}/${MONTH}

for FILE in `ls ${SFA}` ; do
    RAD=`cat "${RAW}/.(get)(${FILE})(checksum)"`
    SAD=`cat "${SFA}/.(get)(${FILE})(checksum)"`
    [ "${RAD}" != "${SAD}" ] && printf "${FILE}\n${RAD} ${SAD}" 
done
done
FD scan
DET=fardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`

2014-04
2014-05
...
2016-06
F00064387_0000.mdaq.root
ADLER32:08004d0e
 ADLER32:00000001
F00064402_0006.mdaq.root
ADLER32:195ca6c0
 ADLER32:00000001
2016-07
2016-08
2016-09
2016-10
2016-11
2016-12
2017-01
2017-02
ND SCAN
DET=neardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`

2014-04
2014-05
N00058872_0006.mdaq.root
ADLER32:55e0bb27
 ADLER32:893256cb
2014-06
...
2016-05
2016-06
N00064974_0009.mdaq.root
ADLER32:29ab236e
 ADLER32:00000001
2016-07
N00070017_0004.mdaq.root
ADLER32:91f578ea
 ADLER32:00000001
2016-08
N00070249_0012.mdaq.root
ADLER32:936f6c53
 ADLER32:00000001
2016-09
N00070427_0014.mdaq.root
ADLER32:ae0da136
 ADLER32:00000001
2016-10
N00070631_0000.mdaq.root
ADLER32:c7543cb4
 ADLER32:00000001
2016-11
N00070940_0002.mdaq.root
ADLER32:42ac8159
 ADLER32:00000001
2016-12
N00062920_0015.mdaq.root
ADLER32:4380482e
 ADLER32:00000001
2017-01
N00071395_0005.mdaq.root
ADLER32:2a88fd3d
 ADLER32:d6dc5ef3
2017-02
N00071670_0012.mdaq.root
ADLER32:076dc6a2
 ADLER32:00000001

#4 Updated by Arthur Kreymer over 3 years ago

  • Due date changed from 04/04/2017 to 04/14/2017
  • Status changed from New to Work in progress
  • % Done changed from 10 to 80

vault_monthly has been updated to vault_monthly.20170401 - just run neardet_data

vaultsfa has been updated to vaultsfa.20170408

  • changed dd to kerberized ftp, using keytab in /opt/minosraw
  • added debug option
  • detailed file list and timing in .err
  • moved logs from nusoft to minos/data/web/vault

The minosraw crontab entry has been enabled on minos-data,
and commited to CVS ( contrib/admin/crontab )

All missing files have been vaulted

Missing files :

DET=fardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`

for MONTH in ${MONTHS} ; do 
    RAW=/pnfs/minos/${DET}/${MONTH}
    SFA=/pnfs/minos/sfa/vault/${DET}/${MONTH}
    NR=`ls ${RAW} | wc -w`
    NS=`ls ${SFA} | wc -w`
    (( NEED = NR - NS ))
    [ "${NR}" != "${NS}" ] && printf "${MONTH} ${NEED}/${NR}\n" 
done

2016-06 678/796

DET=neardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`
2016-08 2433/2575
2016-09 1994/2063
2016-10 2523/2523
2016-11 2473/2499
2016-12 2893/2965
2017-01 3054/3084
2017-02 2612/2682


DETAILS OF CATCHUP
./vaultsfa.20170408 fardet_data 2016-02 echo
./vaultsfa neardet_data  2014-07 
./vaultsfa neardet_data  2014-11 
./vaultsfa neardet_data  2015-07 
./vaultsfa neardet_data  2016-02 
./vaultsfa neardet_data  2016-03 

DET=fardet_data
MOS='2016-06'

set nohup ; { for MO in ${MOS} ; do ./vaultsfa ${DET} ${MO} ; done ; } &

DET=neardet_data
MOS='
2016-08
2016-09
2016-10
2016-11
2016-12
2017-01
2017-02
2017-03'

date
set nohup ; { for MO in ${MOS} ; do ./vaultsfa ${DET} ${MO} ; done ; } &

#5 Updated by Arthur Kreymer over 3 years ago

  • % Done changed from 80 to 90

vault ran under minosraw@minos-data crontab 04/03, successfully

11 20 3 * * ${HOME}/vault_monthly

See http://minos.fnal.gov/vault/neardet_data/

Will close this Issue when vault_monthly and vault are committed to CVS.

#6 Updated by Arthur Kreymer over 3 years ago

  • Status changed from Work in progress to Resolved
  • % Done changed from 90 to 100

Committed vault_monthly and vaultsfa to CVS admin/archiver.

#7 Updated by Arthur Kreymer over 3 years ago

  • Status changed from Resolved to Work in progress

June and July cron jobs archiving May and June terminated with a failed ftp.
Run OK by hand later.
Updated diagnostics. Will see how the Aug 3 run goes.

#8 Updated by Arthur Kreymer 2 months ago

  • Status changed from Work in progress to Closed

The vault script ran properly in Aug 2017,
and through the end of Minerva data taking in 2019.
Marking this issue Closed.

Also available in: Atom PDF