Project

General

Profile

Support #16006

change vault dd to ftp

Added by Arthur Kreymer almost 3 years ago. Updated over 2 years ago.

Status:
Work in progress
Priority:
Low
Start date:
03/28/2017
Due date:
04/14/2017
% Done:

100%

Estimated time:
10.00 h
Duration: 18

Description

Raw data files are copied monthly to /pnfs/minos/sfa/vault
These copies have been partial starting with 2016-06.

o Move the scripts from minosraw@minos-slf6 to minos-data
o Change the copy method to ifdh cp
o Repeal and replace the invalid files

History

#1 Updated by Arthur Kreymer almost 3 years ago


    REMOVE THE EMPTY ARCHIVE FILES

find /pnfs/minos/sfa/vault/neardet_data/2016* -size 0 -exec ls -l {} \;

-rw-r--r-- 1 minosraw e875 0 Jul  3  2016 /pnfs/minos/sfa/vault/neardet_data/2016-06/N00064974_0009.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Aug  3  2016 /pnfs/minos/sfa/vault/neardet_data/2016-07/N00070017_0004.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Sep  3  2016 /pnfs/minos/sfa/vault/neardet_data/2016-08/N00070249_0012.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Oct  3 20:12 /pnfs/minos/sfa/vault/neardet_data/2016-09/N00070427_0014.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Nov  3 20:11 /pnfs/minos/sfa/vault/neardet_data/2016-10/N00070631_0000.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Dec  3 20:12 /pnfs/minos/sfa/vault/neardet_data/2016-11/N00070940_0002.mdaq.root
-rw-r--r-- 1 minosraw e875 0 Jan  3 20:12 /pnfs/minos/sfa/vault/neardet_data/2016-12/N00062920_0015.mdaq.root

find /pnfs/minos/sfa/vault/neardet_data/2016* -size 0 -exec echo "rm -I {}" \;

date
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-06/N00064974_0009.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-07/N00070017_0004.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-08/N00070249_0012.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-09/N00070427_0014.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-10/N00070631_0000.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-11/N00070940_0002.mdaq.root
rm -I /pnfs/minos/sfa/vault/neardet_data/2016-12/N00062920_0015.mdaq.root
date

cut/pasted these

Sat Apr  1 09:45:31 CDT 2017
Sat Apr  1 09:45:31 CDT 2017

find /pnfs/minos/sfa/vault/neardet_data/2017* -size 0 -exec ls -l {} \;
-rw-r--r-- 1 minosraw e875 0 Mar  3 21:01 /pnfs/minos/sfa/vault/neardet_data/2017-02/N00071670_0012.mdaq.root

find /pnfs/minos/sfa/vault/neardet_data/2017* -size 0 -exec echo "rm -I {}" \;

rm -I /pnfs/minos/sfa/vault/neardet_data/2017-02/N00071670_0012.mdaq.root
date
Sat Apr  1 09:47:38 CDT 2017

   Removed the partial file found in the previous scan

ls -l /pnfs/minos/neardet_data/2017-01/N00071395_0005.mdaq.root \
      /pnfs/minos/sfa/vault/neardet_data/2017-01/N00071395_0005.mdaq.root 

-rw-rw-r-- 1 minosraw e875 58803266 Jan  1 07:37 /pnfs/minos/neardet_data/2017-01/N00071395_0005.mdaq.root
-rw-r--r-- 1 minosraw e875 10551296 Feb  3 20:11 /pnfs/minos/sfa/vault/neardet_data/2017-01/N00071395_0005.mdaq.roo

rm -I /pnfs/minos/sfa/vault/neardet_data/2017-01/N00071395_0005.mdaq.root
date
Sat Apr  1 09:51:48 CDT 2017

    REMOVE THE 2014-05 corrupted file

ls -l /pnfs/minos/neardet_data/2014-05/N00058872_0006.mdaq.root \
      /pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root 

-rw-r--r-- 1 minosraw e875 133454372 May 17  2014 /pnfs/minos/neardet_data/2014-05/N00058872_0006.mdaq.root
-rw-r--r-- 1 minosraw e875 133454372 Nov 10  2014 /pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.ro

RAW > . ./setups.sh
RAW > setup encp -q stken

BAS=/pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root
BAD=/pnfs/minos/BAD/neardet_data/sfa_N00058872_0006.mdaq.root

date ; enmv ${BAS} ${BAD}
Sat Apr  1 10:00:51 CDT 2017
ERROR: OS ERROR Unable to access file /pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root: [Errno 13] Permission denied: '/pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root'

   shift it in place ?

BAS=/pnfs/minos/sfa/vault/neardet_data/2014-05/N00058872_0006.mdaq.root
BAD=/pnfs/minos/sfa/vault/neardet_data/2014-05/BAD_N00058872_0006.mdaq.root

date ; enmv ${BAS} ${BAD}
Sat Apr  1 10:03:50 CDT 2017

    TRY IN sfa/vault

mkdir /pnfs/minos/sfa/vault/BAD

BAS=/pnfs/minos/sfa/vault/neardet_data/2014-05/BAD_N00058872_0006.mdaq.root
BAD=/pnfs/minos/sfa/vault/BAD/N00058872_0006.mdaq.root

date ; enmv ${BAS} ${BAD}
Sat Apr  1 10:10:52 CDT 2017

#2 Updated by Arthur Kreymer almost 3 years ago

Will leave this activity in the minosraw account to allow direct mkdir.
Got /opt/minosraw directories for keytabs created with RITM0546594 04/03 .

scp mindata@minos-nearline:/opt/mindata/kt /opt/minosraw/daqdcp-nd.keytab

#3 Updated by Arthur Kreymer almost 3 years ago

Scan of missing vault files, done on March 26

for MONTH in ${MONTHS} ; do
    echo ${MONTH}

RAW=/pnfs/minos/${DET}/${MONTH}
SFA=/pnfs/minos/sfa/vault/${DET}/${MONTH}

for FILE in `ls ${SFA}` ; do
    RAD=`cat "${RAW}/.(get)(${FILE})(checksum)"`
    SAD=`cat "${SFA}/.(get)(${FILE})(checksum)"`
    [ "${RAD}" != "${SAD}" ] && printf "${FILE}\n${RAD} ${SAD}" 
done
done
FD scan
DET=fardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`

2014-04
2014-05
...
2016-06
F00064387_0000.mdaq.root
ADLER32:08004d0e
 ADLER32:00000001
F00064402_0006.mdaq.root
ADLER32:195ca6c0
 ADLER32:00000001
2016-07
2016-08
2016-09
2016-10
2016-11
2016-12
2017-01
2017-02
ND SCAN
DET=neardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`

2014-04
2014-05
N00058872_0006.mdaq.root
ADLER32:55e0bb27
 ADLER32:893256cb
2014-06
...
2016-05
2016-06
N00064974_0009.mdaq.root
ADLER32:29ab236e
 ADLER32:00000001
2016-07
N00070017_0004.mdaq.root
ADLER32:91f578ea
 ADLER32:00000001
2016-08
N00070249_0012.mdaq.root
ADLER32:936f6c53
 ADLER32:00000001
2016-09
N00070427_0014.mdaq.root
ADLER32:ae0da136
 ADLER32:00000001
2016-10
N00070631_0000.mdaq.root
ADLER32:c7543cb4
 ADLER32:00000001
2016-11
N00070940_0002.mdaq.root
ADLER32:42ac8159
 ADLER32:00000001
2016-12
N00062920_0015.mdaq.root
ADLER32:4380482e
 ADLER32:00000001
2017-01
N00071395_0005.mdaq.root
ADLER32:2a88fd3d
 ADLER32:d6dc5ef3
2017-02
N00071670_0012.mdaq.root
ADLER32:076dc6a2
 ADLER32:00000001

#4 Updated by Arthur Kreymer almost 3 years ago

  • Due date changed from 04/04/2017 to 04/14/2017
  • Status changed from New to Work in progress
  • % Done changed from 10 to 80

vault_monthly has been updated to vault_monthly.20170401 - just run neardet_data

vaultsfa has been updated to vaultsfa.20170408

  • changed dd to kerberized ftp, using keytab in /opt/minosraw
  • added debug option
  • detailed file list and timing in .err
  • moved logs from nusoft to minos/data/web/vault

The minosraw crontab entry has been enabled on minos-data,
and commited to CVS ( contrib/admin/crontab )

All missing files have been vaulted

Missing files :

DET=fardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`

for MONTH in ${MONTHS} ; do 
    RAW=/pnfs/minos/${DET}/${MONTH}
    SFA=/pnfs/minos/sfa/vault/${DET}/${MONTH}
    NR=`ls ${RAW} | wc -w`
    NS=`ls ${SFA} | wc -w`
    (( NEED = NR - NS ))
    [ "${NR}" != "${NS}" ] && printf "${MONTH} ${NEED}/${NR}\n" 
done

2016-06 678/796

DET=neardet_data
MONTHS=`ls /pnfs/minos/sfa/vault/${DET}`
2016-08 2433/2575
2016-09 1994/2063
2016-10 2523/2523
2016-11 2473/2499
2016-12 2893/2965
2017-01 3054/3084
2017-02 2612/2682


DETAILS OF CATCHUP
./vaultsfa.20170408 fardet_data 2016-02 echo
./vaultsfa neardet_data  2014-07 
./vaultsfa neardet_data  2014-11 
./vaultsfa neardet_data  2015-07 
./vaultsfa neardet_data  2016-02 
./vaultsfa neardet_data  2016-03 

DET=fardet_data
MOS='2016-06'

set nohup ; { for MO in ${MOS} ; do ./vaultsfa ${DET} ${MO} ; done ; } &

DET=neardet_data
MOS='
2016-08
2016-09
2016-10
2016-11
2016-12
2017-01
2017-02
2017-03'

date
set nohup ; { for MO in ${MOS} ; do ./vaultsfa ${DET} ${MO} ; done ; } &

#5 Updated by Arthur Kreymer over 2 years ago

  • % Done changed from 80 to 90

vault ran under minosraw@minos-data crontab 04/03, successfully

11 20 3 * * ${HOME}/vault_monthly

See http://minos.fnal.gov/vault/neardet_data/

Will close this Issue when vault_monthly and vault are committed to CVS.

#6 Updated by Arthur Kreymer over 2 years ago

  • Status changed from Work in progress to Resolved
  • % Done changed from 90 to 100

Committed vault_monthly and vaultsfa to CVS admin/archiver.

#7 Updated by Arthur Kreymer over 2 years ago

  • Status changed from Resolved to Work in progress

June and July cron jobs archiving May and June terminated with a failed ftp.
Run OK by hand later.
Updated diagnostics. Will see how the Aug 3 run goes.



Also available in: Atom PDF