Feature #5209
report dataset state on dcache disk
Description
Prepare a prototype tool to measure quickly
what part of a given SAM dataset resides on DCache disk.
History
#1 Updated by Arthur Kreymer almost 6 years ago
Items done so far :
o tested dccp -P -t -1
o found files to test with
off disk reco_near r1.12/r1/14 38/832
on disk fardet_data/2001-10 2001-11 55/491
o got minos dimension list to guide queries
o made testing datasets
sntp-near-r1.12
sntp-near-r1.14
fardet-2001-10
fardet-2001-11
daikon0farN100
daikon0farNall
o manually assessed small datasets
o tested job status query techniques
o filled FN array with file names from the dataset
TO DO -
x run the queries in parallel
x collate and report the results
x wrap it all in a script
#2 Updated by Arthur Kreymer almost 6 years ago
- % Done changed from 50 to 80
o made DCPING <FILE> function to echo the disk state
o extended DCPING N to look up file from the array
o made an ENTAIL wrapper to execute DCPING serially for initial tests
o expanded ENTAIL to run N threads in the background with direct output
o did initial timings
only getting 15 tests/second with 100 threads
bare speed is 5 tests/second
#3 Updated by Arthur Kreymer almost 6 years ago
Created user level script.
Messages go to stderr
File status goes to stdout
/grid/fermiapp/common/tools/test/dcdisk
dcdisk prototype 20140117 - could change in name and function without warningUsage : dcdisk experiment samdataset
53 in fardet-2001-10
TIMING TESTS ON MINOS27
time /grid/fermiapp/common/tools/test/dcdisk minos fardet-2001-10 > /dev/null
53 in fardet-2001-10
real 0m15.428s
user 0m4.241s
sys 0m15.445s
time /grid/fermiapp/common/tools/test/dcdisk minos fardet-2001-11 > /dev/null
445 in fardet-2001-11
real 0m48.141s
user 0m37.164s
sys 2m8.656s
time /grid/fermiapp/common/tools/test/dcdisk minos sntp-near-r1.12 > /dev/null
31 in sntp-near-r1.12
real 0m14.743s
user 0m2.480s
sys 0m8.846s
time /grid/fermiapp/common/tools/test/dcdisk minos sntp-near-r1.14 > /dev/null
832 in sntp-near-r1.14
real 1m29.339s
user 1m10.055s
sys 4m12.783s
#4 Updated by Arthur Kreymer almost 6 years ago
On Friday Jan 17, deployed a dcdisk test script.
/grid/fermiapp/common/tools/test/dcdisk
stdout gets a list of files prefixed by 0 if on disk, something else if not.
stderr gets summary messages, like
31 in sntp-near-r1.12
Presently this runs 100 threads
----------------------------------------------------
MINOS27 > /grid/fermiapp/common/tools/test/dcdisk
dcdisk prototype 20140117 - could change in name and function without warningUsage : dcdisk experiment samdataset
-----------------------------------------------------
MINOS27 > time /grid/fermiapp/common/tools/test/dcdisk minos sntp-near-r1.14 > /var/tmp/r114files
dcdisk prototype 20140117 - could change in name and function without warning832 in sntp-near-r1.14
real 1m23.772s
user 1m10.144s
sys 4m0.865s
MINOS27 > head /var/tmp/r114files
255 N00005845_0009.spill.sntp.R1.14.root
255 N00005845_0012.spill.sntp.R1.14.root
255 N00005812_0000.spill.sntp.R1.14.root
255 N00005833_0000.spill.sntp.R1.14.root
255 N00005845_0001.spill.sntp.R1.14.root
255 N00005834_0000.spill.sntp.R1.14.root
255 N00005845_0026.spill.sntp.R1.14.root
255 N00005845_0015.spill.sntp.R1.14.root
255 N00005845_0045.spill.sntp.R1.14.root
255 N00005882_0001.spill.sntp.R1.14.root
And a dataset on disk is likeMINOS27 > head -4 /var/tmp/far10
0 F00000911_0000.mdaq.root
0 F00000989_0000.mdaq.root
0 F00000901_0000.mdaq.root
0 F00001000_0000.mdaq.root
#5 Updated by Arthur Kreymer almost 6 years ago
- % Done changed from 80 to 90
#6 Updated by Arthur Kreymer almost 6 years ago
Revised dcdisk script 2014 01 23
exported SAM_EXPERIMENT
cleared DCACH_IO_TUNNEL to disable kerberos
added file count on disk to stderr
align file count printouts