Project

General

Profile

Feature #5209

report dataset state on dcache disk

Added by Arthur Kreymer over 5 years ago. Updated over 5 years ago.

Status:
New
Priority:
Urgent
Target version:
-
Start date:
01/16/2014
Due date:
01/17/2014
% Done:

90%

Estimated time:
8.00 h
Spent time:
Duration: 2

Description

Prepare a prototype tool to measure quickly
what part of a given SAM dataset resides on DCache disk.

History

#1 Updated by Arthur Kreymer over 5 years ago

Items done so far :
o tested dccp -P -t -1

o found files to test with
off disk reco_near r1.12/r1/14 38/832
on disk fardet_data/2001-10 2001-11 55/491

o got minos dimension list to guide queries

o made testing datasets
sntp-near-r1.12
sntp-near-r1.14
fardet-2001-10
fardet-2001-11
daikon0farN100
daikon0farNall

o manually assessed small datasets

o tested job status query techniques

o filled FN array with file names from the dataset

TO DO -

x run the queries in parallel
x collate and report the results
x wrap it all in a script

#2 Updated by Arthur Kreymer over 5 years ago

  • % Done changed from 50 to 80

o made DCPING <FILE> function to echo the disk state
o extended DCPING N to look up file from the array
o made an ENTAIL wrapper to execute DCPING serially for initial tests
o expanded ENTAIL to run N threads in the background with direct output
o did initial timings
only getting 15 tests/second with 100 threads
bare speed is 5 tests/second

#3 Updated by Arthur Kreymer over 5 years ago

Created user level script.
Messages go to stderr
File status goes to stdout

/grid/fermiapp/common/tools/test/dcdisk

dcdisk prototype 20140117 - could change in name and function without warning

Usage : dcdisk experiment samdataset

53 in fardet-2001-10

TIMING TESTS ON MINOS27

time /grid/fermiapp/common/tools/test/dcdisk minos fardet-2001-10 > /dev/null
53 in fardet-2001-10
real 0m15.428s
user 0m4.241s
sys 0m15.445s

time /grid/fermiapp/common/tools/test/dcdisk minos fardet-2001-11 > /dev/null
445 in fardet-2001-11
real 0m48.141s
user 0m37.164s
sys 2m8.656s

time /grid/fermiapp/common/tools/test/dcdisk minos sntp-near-r1.12 > /dev/null
31 in sntp-near-r1.12
real 0m14.743s
user 0m2.480s
sys 0m8.846s

time /grid/fermiapp/common/tools/test/dcdisk minos sntp-near-r1.14 > /dev/null
832 in sntp-near-r1.14
real 1m29.339s
user 1m10.055s
sys 4m12.783s

#4 Updated by Arthur Kreymer over 5 years ago

On Friday Jan 17, deployed a dcdisk test script.

/grid/fermiapp/common/tools/test/dcdisk

stdout gets a list of files prefixed by 0 if on disk, something else if not.
stderr gets summary messages, like
31 in sntp-near-r1.12

Presently this runs 100 threads
----------------------------------------------------
MINOS27 > /grid/fermiapp/common/tools/test/dcdisk

dcdisk prototype 20140117 - could change in name and function without warning

Usage : dcdisk experiment samdataset
-----------------------------------------------------
MINOS27 > time /grid/fermiapp/common/tools/test/dcdisk minos sntp-near-r1.14 > /var/tmp/r114files

dcdisk prototype 20140117 - could change in name and function without warning

832 in sntp-near-r1.14

real 1m23.772s
user 1m10.144s
sys 4m0.865s

MINOS27 > head /var/tmp/r114files
255 N00005845_0009.spill.sntp.R1.14.root
255 N00005845_0012.spill.sntp.R1.14.root
255 N00005812_0000.spill.sntp.R1.14.root
255 N00005833_0000.spill.sntp.R1.14.root
255 N00005845_0001.spill.sntp.R1.14.root
255 N00005834_0000.spill.sntp.R1.14.root
255 N00005845_0026.spill.sntp.R1.14.root
255 N00005845_0015.spill.sntp.R1.14.root
255 N00005845_0045.spill.sntp.R1.14.root
255 N00005882_0001.spill.sntp.R1.14.root

And a dataset on disk is like

MINOS27 > head -4 /var/tmp/far10
0 F00000911_0000.mdaq.root
0 F00000989_0000.mdaq.root
0 F00000901_0000.mdaq.root
0 F00001000_0000.mdaq.root

#5 Updated by Arthur Kreymer over 5 years ago

  • % Done changed from 80 to 90

#6 Updated by Arthur Kreymer over 5 years ago

Revised dcdisk script 2014 01 23

exported SAM_EXPERIMENT
cleared DCACH_IO_TUNNEL to disable kerberos
added file count on disk to stderr
align file count printouts



Also available in: Atom PDF