Project

General

Profile

Support #20862

annie-daq02 cleanup for Sep 2018 reinstall

Added by Arthur Kreymer about 1 year ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Start date:
09/17/2018
Due date:
09/21/2018
% Done:

90%

Estimated time:
4.00 h
Spent time:
Duration: 5

Description

We should remove archived data files,
and make a safe copy of the annie home area and select data logs
preparing for a reinstall of annie-daq02.

This is a partial rehearsal for annie-daq01.

History

#1 Updated by Arthur Kreymer about 1 year ago

  • % Done changed from 0 to 40
  • Estimated time changed from 2.00 h to 4.00 h

Planning for archival of data files from anniedaq2:/data
Per slack contact
2.8 TB roughly

ssh

ssh annie-daq02

$ du -sm *
2379862 archive
0 images
1 logs
1 output
59 postgres
491639 temp_29-05-2016

cd archive
ls *.root | wc -l
3315

Some files in /data/temp are identical to those in /data/archive

TDIR=/data/temp_29-05-2016
ADIR=/data/archive

ls ${TDIR} | wc -l
675

TFILES=`ls ${TDIR}`
for TFILE in ${TFILES} ; do
ls -l ${TDIR}/${TFILE} ${ADIR}/${TFILE}
done | less

{
date
for TFILE in ${TFILES} ; do
printf "${TFILE}\n"
diff ${TDIR}/${TFILE} ${ADIR}/${TFILE}
done
date ; } 2>&1 | tee /tmp/tdiffs

Mon Aug 6 13:36:43 CDT 2018
DataR15S0p0T0May__2.root
DataR15S0p10T0May__2.root
...
DataR29S2p13T0May_14_21:29.root
DataR29S2p14T0May_14_21:37.root
DataR29S2p1T0May_14_19:48.root
Mon Aug 6 14:37:18 CDT 2018

date ; rm -r /data/temp_29-05-2016
Mon Aug 6 14:56:51 CDT 2018

FILES=`ls /data/archive`

for FILE in ${FILES} ; do
FS=`ls -l /data/archive/${FILE} | tr -s ' ' | cut -f 5 -d ' '`
printf "${FILE} ${FS}\n"
done 2>&1 | tee /tmp/filesiz

wc -l /tmp/filesiz
3315 /tmp/filesiz

scp /tmp/filesiz anniepro@annie01:anniedaq02/filesiz


cat filesiz | while read LINE ; do

FILE=`echo ${LINE} | cut -f 1 -d ' ' | tr -d ":"`
SIZE=`echo ${LINE} | cut -f 2 -d ' '`
RUNN=`echo ${FILE} | cut -f 2 -d 'R' | cut -f 1 -d 'S'`
RUN=`printf "%4.4d" ${RUNN}`
PSIZ=`ls -l /pnfs/annie/raw/data/${RUN}/${FILE} | tr -s ' ' | cut -f 5 -d ' '`
[ ${SIZE} -ne ${PSIZ} ] && echo ${RUN} ${FILE} ${SIZE} ${PSIZ}
done

#2 Updated by Arthur Kreymer about 1 year ago

I would remove the archived /data/archive files, but cannot log into annie-gw01 right now.
The system seems to be off the network.

#3 Updated by Arthur Kreymer about 1 year ago

  • % Done changed from 40 to 60

I re-verified that /data/*.root files match PNFS in size.
I removed the /data/*.root files.

ssh anniepro@annie01
cd anniedaq02

NF=0
cat filesiz | while read LINE ; do
(( NF++ ))
FILE=`echo ${LINE} | cut -f 1 -d ' ' | tr -d ":"`
printf "\r${NF} ${FILE}"
SIZE=`echo ${LINE} | cut -f 2 -d ' '`
RUNN=`echo ${FILE} | cut -f 2 -d 'R' | cut -f 1 -d 'S'`
RUN=`printf "%4.4d" ${RUNN}`
PSIZ=`ls -l /pnfs/annie/raw/data/${RUN}/${FILE} | tr -s ' ' | cut -f 5 -d ' '`
[ ${SIZE} -ne ${PSIZ} ] && echo ${RUN} ${FILE} ${SIZE} ${PSIZ}
done

DataR41S1p31T0May_23_1852.root^C

on anniedaq02

date ; rm /data/archive/*.root
Thu Sep 20 16:07:34 CDT 2018

[annie@annie-daq02 ~]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 2.5G 16G 14% /
tmpfs 3.9G 0 3.9G 0% /dev/shm
/dev/sda6 873G 6.4G 822G 1% /home
/dev/sda3 4.7G 11M 4.5G 1% /tmp
/dev/sda5 4.7G 855M 3.7G 19% /var
/dev/sdb1 9.1T 92M 9.1T 1% /data

#4 Updated by Arthur Kreymer about 1 year ago

  • % Done changed from 60 to 90

annieraw@annie-daq02

DAQ2=anniepro@annie01:/annie/data/users/anniepro/maint/annie-daq02

scp -r /home/annieraw ${DAQ2}/home/annieraw

annie@annie-daq02

scp -r /data ${DAQ2}/data

du -sm /data
117 /data

find /data | wc -l
2104

root@annie-daq02

scp -r /home/annie ${DAQ2}/home/annie

scp -r /home/vmeroot ${DAQ2}/home/vmeroot

find /home/annie -type f | wc -l
80711

find /home/annie | wc -l
88019

cd /home
tar -cf /data/home.tar .

scp /data/home.tar ${DAQ2}/data/home.tar



Also available in: Atom PDF