Project

General

Profile

Support #5603

Declare recent MC sntp to SAM

Added by Arthur Kreymer almost 7 years ago. Updated almost 7 years ago.

Status:
Assigned
Priority:
High
Assignee:
Manungu Kiveni
Start date:
03/06/2014
Due date:
03/26/2014
% Done:

60%

Estimated time:
8.00 h
Spent time:
12.10 h
Duration: 21

Description

Recent MC sntp have not been declared to SAM.
1) Fix this for future processing.
2) Declare these to SAM post-hoc

History

#1 Updated by Arthur Kreymer almost 7 years ago

  • Status changed from New to Assigned
  • Estimated time set to 4.00 h

I will extract SAM code from the roundup process
to do the post-hoc declarations.

I will do this on a few files,
then hand this to someone else for bulk running.

Which files are involved ?

#2 Updated by Arthur Kreymer almost 7 years ago

  • % Done changed from 0 to 20
  • Estimated time changed from 4.00 h to 8.00 h

Reviewed ~kreymer/minos/LOG, found 2011 script samcout
previously used to catch up on MC declarations.

Evaluating with
MDO=/pnfs/minos/mcout_data/elm3/near/daikon_10/M000200N_NOVA/sntp_data

GET THE SCRIPT
cd /grid/fermiapp/minos/minospro/scripts
cp ADMIN/minfarm/samcout samcout
GET THE SAM access file
mkdir ${HOME}/.grid
chmod 700 ${HOME}/.grid
scp minfarm@minos27:.grid/samdbs_prd ${HOME}/.grid/
PRO > ./samcout elm3 near daikon_10 M000200N_NOVA

/pnfs/minos/mcout_data/elm3/near/daikon_10/M000200N_NOVA

cand_data mrnt_data sntp_data
cand_data 0
mrnt_data 327
sntp_data 327

PRO > ./samcout elm3 near daikon_10 M000200N_NOVA verify
Needed 654
Verified 654

LOG 
rw-r--r- 1 minospro e875 7852035 Mar 22 10:02 /var/tmp/samcout.log
STARTED   Sat Mar 22 14:58:14 2014
STARTED   Sat Mar 22 14:58:14 2014
FINISHED  Sat Mar 22 15:02:42 2014

#3 Updated by Arthur Kreymer almost 7 years ago

Still need to do :
o see why roundup is not declaring files, and fix this
o test samcout declare on sample of files
o inventory existing files, and apply samcout

#4 Updated by Arthur Kreymer almost 7 years ago

  • Due date set to 03/26/2014

#5 Updated by Arthur Kreymer almost 7 years ago

I found the roundup vs saddreco incompatibility last weekend.
roundup is running, for MC, saddreco -w /minos/data/...
but saddreco had not implemented the -w option.

I have a new version which implements -w, about to be tested.
Will try to roll this into production tomorrow.

#6 Updated by Arthur Kreymer almost 7 years ago

Tested a revised saddreco with
./samcout elm3 near daikon_10 M000200N_NOVA declare

This worked, but is still verbose, printing metadata for each file.

#7 Updated by Arthur Kreymer almost 7 years ago

Deployed the revised saddreco in production.
First test that nightly reco adds correctly to sam.
The see that simulation is OK, using samcout.

Want to get this in place before more MC processing runs.

#8 Updated by Arthur Kreymer almost 7 years ago

Details of deployment .

minospro@minos27

cd /grid/fermiapp/minos/minospro/scripts/

cp -a saddreco saddreco.20120920

ln -sf saddreco.20140325 saddreco
date

Wed Apr 2 18:25:30 CDT 2014

#9 Updated by Arthur Kreymer almost 7 years ago

Roundup picked up mid-day today,
catching up on SAM declarations for raw data,
after the saddreco symlink correction.
There was no need for a special manual run of saddreco.

#10 Updated by Arthur Kreymer almost 7 years ago

Ran this test Apr 5.

minospro@minos27

./samcout elm3 far daikon_10 M000200N_NOVA verify
cand_data mrnt_data sntp_data
RECODIR /pnfs/minos/mcout_data/elm3/far/daikon_10
DETECT far
MCREL daikon_10
RELEASE elm3
MONTH M000200N_NOVA
BAIL 999999
there is directory /pnfs/minos/mcout_data/elm3/far/daikon_10
RECODIRS ['M000200N_NOVA/mrnt_data/100', 'M000200N_NOVA/mrnt_data/101', 'M000200N_NOVA/mrnt_data/102', 'M000200N_NOVA/mrnt_data/103', 'M000200N_NOVA/mrnt_data/104', 'M000200N_NOVA/mrnt_data/105', 'M000200N_NOVA/mrnt_data/106', 'M000200N_NOVA/mrnt_data/107', 'M000200N_NOVA/mrnt_data/108', 'M000200N_NOVA/sntp_data/100', 'M000200N_NOVA/sntp_data/101', 'M000200N_NOVA/sntp_data/102', 'M000200N_NOVA/sntp_data/103', 'M000200N_NOVA/sntp_data/104', 'M000200N_NOVA/sntp_data/105', 'M000200N_NOVA/sntp_data/106', 'M000200N_NOVA/sntp_data/107', 'M000200N_NOVA/sntp_data/108']
STARTED Sat Apr 5 15:24:09 2014
saddreco 20140405
Declaring to SAM prd far elm3 M000200N_NOVA verify
Needed /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100
OK - verified f21431008_0026_M000200N_D10_NOVA.mrnt.elm3.0.root /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100(vp8927.640)
OK - verified f21031006_0000_M000200N_D10_NOVA.mrnt.elm3.0.root /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100(vp8927.10)
...
TARTED Sat Apr 5 15:24:09 2014
FINISHED Sat Apr 5 15:34:31 2014

SUMMARY 
Needed  /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/100
Needed  116 files, Rate was  2.535
Needed  /pnfs/minos/mcout_data/elm3/far/daikon_10/M000200N_NOVA/mrnt_data/101
...
    RATES 
Needed  116 files, Rate was  2.535
Needed   79 files, Rate was  2.206
...
Needed   24 files, Rate was  1.758
Needed   1262
  Verified 1262
LOG 
rw-r--r- 1 minospro e875 190665 Apr  5 10:34 /var/tmp/samcout.log
STARTED   Sat Apr  5 15:24:09 2014
STARTED   Sat Apr  5 15:24:09 2014
FINISHED  Sat Apr  5 15:34:31 2014

#11 Updated by Arthur Kreymer almost 7 years ago

Log into minospro@minos27

SAMCOUT=/grid/fermiapp/minos/minospro/scripts/samcout

Get a file count from
${SAMCOUT} <reco> <det> <mcver> <mcconfig> 
Verify metadata
${SAMCOUT} <reco> <det> <mcver> <mcconfig>  verify
Declare
${SAMCOUT} <reco> <det> <mcver> <mcconfig>  declare

#12 Updated by Arthur Kreymer almost 7 years ago

Joseph will use samcout to survey what files need to be declared.
After the survey, he will declare these to SAM.

We expect that everything produced since 2013 is needed.

#13 Updated by Arthur Kreymer almost 7 years ago

  • % Done changed from 20 to 60

Posted in error

#14 Updated by Arthur Kreymer almost 7 years ago

Posted in error

#15 Updated by Arthur Kreymer almost 7 years ago

Posted in error

#16 Updated by Arthur Kreymer almost 7 years ago

  • Assignee changed from Arthur Kreymer to Manungu Kiveni

Joseph - any progress on this ?

Also available in: Atom PDF