Support #5603
Declare recent MC sntp to SAM
Description
Recent MC sntp have not been declared to SAM.
1) Fix this for future processing.
2) Declare these to SAM post-hoc
History
#1 Updated by Arthur Kreymer almost 6 years ago
- Status changed from New to Assigned
- Estimated time set to 4.00 h
I will extract SAM code from the roundup process
to do the post-hoc declarations.
I will do this on a few files,
then hand this to someone else for bulk running.
Which files are involved ?
#2 Updated by Arthur Kreymer over 5 years ago
- % Done changed from 0 to 20
- Estimated time changed from 4.00 h to 8.00 h
Reviewed ~kreymer/minos/LOG, found 2011 script samcout
previously used to catch up on MC declarations.
Evaluating with
MDO=/pnfs/minos/mcout_data/elm3/near/daikon_10/M000200N_NOVA/sntp_data
GET THE SCRIPT
cd /grid/fermiapp/minos/minospro/scripts
cp ADMIN/minfarm/samcout samcoutGET THE SAM access file
mkdir ${HOME}/.grid
chmod 700 ${HOME}/.grid
scp minfarm@minos27:.grid/samdbs_prd ${HOME}/.grid/
PRO > ./samcout elm3 near daikon_10 M000200N_NOVA/pnfs/minos/mcout_data/elm3/near/daikon_10/M000200N_NOVA
cand_data mrnt_data sntp_data
cand_data 0
mrnt_data 327
sntp_data 327
PRO > ./samcout elm3 near daikon_10 M000200N_NOVA verify
Needed 654
Verified 654
LOG
rw-r--r- 1 minospro e875 7852035 Mar 22 10:02 /var/tmp/samcout.log
STARTED Sat Mar 22 14:58:14 2014
STARTED Sat Mar 22 14:58:14 2014
FINISHED Sat Mar 22 15:02:42 2014#3 Updated by Arthur Kreymer over 5 years ago
Still need to do :
o see why roundup is not declaring files, and fix this
o test samcout declare on sample of files
o inventory existing files, and apply samcout
#4 Updated by Arthur Kreymer over 5 years ago
- Due date set to 03/26/2014
#5 Updated by Arthur Kreymer over 5 years ago
I found the roundup vs saddreco incompatibility last weekend.
roundup is running, for MC, saddreco -w /minos/data/...
but saddreco had not implemented the -w option.
I have a new version which implements -w, about to be tested.
Will try to roll this into production tomorrow.
#6 Updated by Arthur Kreymer over 5 years ago
Tested a revised saddreco with
./samcout elm3 near daikon_10 M000200N_NOVA declare
This worked, but is still verbose, printing metadata for each file.
#7 Updated by Arthur Kreymer over 5 years ago
Deployed the revised saddreco in production.
First test that nightly reco adds correctly to sam.
The see that simulation is OK, using samcout.
Want to get this in place before more MC processing runs.
#8 Updated by Arthur Kreymer over 5 years ago
Details of deployment .
minospro@minos27
cd /grid/fermiapp/minos/minospro/scripts/
cp -a saddreco saddreco.20120920
ln -sf saddreco.20140325 saddreco
date
Wed Apr 2 18:25:30 CDT 2014
#9 Updated by Arthur Kreymer over 5 years ago
Roundup picked up mid-day today,
catching up on SAM declarations for raw data,
after the saddreco symlink correction.
There was no need for a special manual run of saddreco.
#10 Updated by Arthur Kreymer over 5 years ago
Ran this test Apr 5.
minospro@minos27
./samcout elm3 far daikon_10 M000200N_NOVA verify
cand_data mrnt_data sntp_data
RECODIR /pnfs/minos/mcout_data/elm3/far/daikon_10
DETECT far
MCREL daikon_10
RELEASE elm3
MONTH M000200N_NOVA
BAIL 999999
there is directory /pnfs/minos/mcout_data/elm3/far/daikon_10
RECODIRS ['M000200N_NOVA/mrnt_data/100', 'M000200N_NOVA/mrnt_data/101', 'M000200N_NOVA/mrnt_data/102', 'M000200N_NOVA/mrnt_data/103', 'M000200N_NOVA/mrnt_data/104', 'M000200N_NOVA/mrnt_data/105', 'M000200N_NOVA/mrnt_data/106', 'M000200N_NOVA/mrnt_data/107', 'M000200N_NOVA/mrnt_data/108', 'M000200N_NOVA/sntp_data/100', 'M000200N_NOVA/sntp_data/101', 'M000200N_NOVA/sntp_data/102', 'M000200N_NOVA/sntp_data/103', 'M000200N_NOVA/sntp_data/104', 'M000200N_NOVA/sntp_data/105', 'M000200N_NOVA/sntp_data/106', 'M000200N_NOVA/sntp_data/107', 'M000200N_NOVA/sntp_data/108']
STARTED Sat Apr 5 15:24:09 2014
saddreco 20140405
Declaring to SAM prd far elm3 M000200N_NOVA verify
Needed /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100
OK - verified f21431008_0026_M000200N_D10_NOVA.mrnt.elm3.0.root /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100(vp8927.640)
OK - verified f21031006_0000_M000200N_D10_NOVA.mrnt.elm3.0.root /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100(vp8927.10)
...
TARTED Sat Apr 5 15:24:09 2014
FINISHED Sat Apr 5 15:34:31 2014
SUMMARY
Needed /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100
Needed 116 files, Rate was 2.535
Needed /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/101
...
RATES
Needed 116 files, Rate was 2.535
Needed 79 files, Rate was 2.206
...
Needed 24 files, Rate was 1.758Needed 1262
Verified 1262LOG
rw-r--r- 1 minospro e875 190665 Apr 5 10:34 /var/tmp/samcout.log
STARTED Sat Apr 5 15:24:09 2014
STARTED Sat Apr 5 15:24:09 2014
FINISHED Sat Apr 5 15:34:31 2014#11 Updated by Arthur Kreymer over 5 years ago
Log into minospro@minos27
SAMCOUT=/grid/fermiapp/minos/minospro/scripts/samcout
Get a file count from
${SAMCOUT} <reco> <det> <mcver> <mcconfig> Verify metadata
${SAMCOUT} <reco> <det> <mcver> <mcconfig> verifyDeclare
${SAMCOUT} <reco> <det> <mcver> <mcconfig> declare#12 Updated by Arthur Kreymer over 5 years ago
Joseph will use samcout to survey what files need to be declared.
After the survey, he will declare these to SAM.
We expect that everything produced since 2013 is needed.
#14 Updated by Arthur Kreymer over 5 years ago
Posted in error
#15 Updated by Arthur Kreymer over 5 years ago
Posted in error
#16 Updated by Arthur Kreymer over 5 years ago
- Assignee changed from Arthur Kreymer to Manungu Kiveni
Joseph - any progress on this ?