Bug #10763
Retire old EOS nodes (cmseos2-5)
Start date:
11/24/2015
Due date:
% Done:
0%
Estimated time:
(Total: 0.00 h)
Description
The following nodes should be retired: cmseos2,3,4,5. This are all 2x1Gbps nodes and very old.
I got the following email from David Fagan:
These are old
cmseos2,3,4,5 (disk 2009) 36TB
?
cmseos1 (2011) 108TB
-----
Getting there...
cmseos27 (2012) 162TB
For now I'd keep cmseos1 and 27, they both have 10GE NICs.
I think I'll need to write a small procedure on how to migrate disks.
Subtasks
History
#1 Updated by Gerard Bernabeu Altayo over 4 years ago
Drain documentation available at http://eos.readthedocs.org/en/latest/configuration/draining.html
It does not make sense to make our own parallel documentation, I will just post in this ticket what I've done exactly for future reference:
[root@cmssrv222 ~]# eos fs ls -d
#....................................................................................................................................
# host (#...) # id # path # drain # progress # files # bytes-left # timeleft #retry #wopen
#....................................................................................................................................
[root@cmssrv222 ~]# eos fs ls | grep cmseos2
cmseos2.fnal.gov (1095) 74 /storage/data1 default.0 booted rw nodrain online
cmseos2.fnal.gov (1095) 75 /storage/data2 default.1 booted rw nodrain online
cmseos2.fnal.gov (1095) 76 /storage/data3 default.3 booted rw nodrain online
cmseos25.fnal.gov (1095) 103 /storage/data1 default.0 booted rw nodrain online
cmseos25.fnal.gov (1095) 104 /storage/data2 default.1 booted rw nodrain online
cmseos26.fnal.gov (1095) 105 /storage/data1 default.2 booted rw nodrain online
cmseos26.fnal.gov (1095) 106 /storage/data2 default.3 booted rw nodrain online
cmseos27.fnal.gov (1095) 107 /storage/data1 default.0 booted rw nodrain online
cmseos27.fnal.gov (1095) 108 /storage/data2 default.1 booted rw nodrain online
cmseos27.fnal.gov (1095) 109 /storage/data3 default.2 booted rw nodrain online
cmseos28.fnal.gov (1095) 110 /storage/data1 default.0 booted rw nodrain online
cmseos28.fnal.gov (1095) 111 /storage/data2 default.1 booted rw nodrain online
cmseos29.fnal.gov (1095) 112 /storage/data1 default.2 booted rw nodrain online
cmseos29.fnal.gov (1095) 113 /storage/data2 default.3 booted rw nodrain online
[root@cmssrv222 ~]# eos fs config 74 configstatus=drain
[root@cmssrv222 ~]# eos fs config 75 configstatus=drain
[root@cmssrv222 ~]# eos fs config 76 configstatus=drain
[root@cmssrv222 ~]# eos fs ls | grep cmseos3.fnal
cmseos3.fnal.gov (1095) 71 /storage/data1 default.0 booted rw nodrain online
cmseos3.fnal.gov (1095) 72 /storage/data2 default.2 booted rw nodrain online
cmseos3.fnal.gov (1095) 73 /storage/data3 default.1 booted rw nodrain online
[root@cmssrv222 ~]# eos fs config 71 configstatus=drain
[root@cmssrv222 ~]# eos fs config 72 configstatus=drain
[root@cmssrv222 ~]# eos fs config 73 configstatus=drain
[root@cmssrv222 ~]# eos fs ls | grep cmseos4.fnal
cmseos4.fnal.gov (1095) 68 /storage/data1 default.0 booted rw nodrain online
cmseos4.fnal.gov (1095) 69 /storage/data2 default.1 booted rw nodrain online
cmseos4.fnal.gov (1095) 70 /storage/data3 default.2 booted rw nodrain online
[root@cmssrv222 ~]# eos fs config 68 configstatus=drain
[root@cmssrv222 ~]# eos fs config 69 configstatus=drain
[root@cmssrv222 ~]# eos fs config 70 configstatus=drain
[root@cmssrv222 ~]# eos fs ls | grep cmseos5.fnal
cmseos5.fnal.gov (1095) 65 /storage/data1 default.0 booted rw nodrain online
cmseos5.fnal.gov (1095) 66 /storage/data2 default.1 booted rw nodrain online
cmseos5.fnal.gov (1095) 67 /storage/data3 default.2 booted rw nodrain online
[root@cmssrv222 ~]# eos fs config 65 configstatus=drain
[root@cmssrv222 ~]# eos fs config 66 configstatus=drain
[root@cmssrv222 ~]# eos fs config 67 configstatus=drain
[root@cmssrv222 ~]# eos fs ls -d
#....................................................................................................................................
# host (#...) # id # path # drain # progress # files # bytes-left # timeleft #retry #wopen
#....................................................................................................................................
cmseos5.fnal.gov (1095) 65 /storage/data1 prepare 0 0.00 0.00 B 50 0 4
cmseos5.fnal.gov (1095) 66 /storage/data2 prepare 0 0.00 0.00 B 53 0 0
cmseos5.fnal.gov (1095) 67 /storage/data3 prepare 0 0.00 0.00 B 57 0 4
cmseos4.fnal.gov (1095) 68 /storage/data1 prepare 0 0.00 0.00 B 28 0 3
cmseos4.fnal.gov (1095) 69 /storage/data2 prepare 0 0.00 0.00 B 32 0 1
cmseos4.fnal.gov (1095) 70 /storage/data3 prepare 0 0.00 0.00 B 36 0 2
cmseos3.fnal.gov (1095) 71 /storage/data1 prepare 0 0.00 0.00 B 3 0 5
cmseos3.fnal.gov (1095) 72 /storage/data2 prepare 0 0.00 0.00 B 7 0 0
cmseos3.fnal.gov (1095) 73 /storage/data3 prepare 0 0.00 0.00 B 10 0 2
cmseos2.fnal.gov (1095) 74 /storage/data1 draining 0 143.51 k 7.59 TB 86298 0 0
cmseos2.fnal.gov (1095) 75 /storage/data2 draining 0 138.33 k 7.53 TB 86324 0 2
cmseos2.fnal.gov (1095) 76 /storage/data3 draining 0 152.90 k 8.09 TB 86328 0 2
[root@cmssrv222 ~]#
#2 Updated by Gerard Bernabeu Altayo over 4 years ago
- Subject changed from Retire old EOS nodes to Retire old EOS nodes (cmseos2-5)
The drain expired for some nodes, resubmitted it with:
[root@cmssrv222 ~]# for i in `seq 65 76`; do eos fs config $i configstatus=drain; done
#3 Updated by Gerard Bernabeu Altayo over 4 years ago
All nodes have some issues with files, focusing on cmseos5 first I think that there is inevitable data loss on it:
EOS Console [root://localhost] |/eos/> fs ls -d
#....................................................................................................................................
# host (#...) # id # path # drain # progress # files # bytes-left # timeleft #retry #wopen
#....................................................................................................................................
cmseos5.fnal.gov (1095) 65 /storage/data1 stalling 0 190.00 284.26 MB 99999999999 0 0
cmseos5.fnal.gov (1095) 66 /storage/data2 stalling 0 251.00 3.50 GB 99999999999 0 0
cmseos5.fnal.gov (1095) 67 /storage/data3 stalling 0 240.00 2.84 GB 99999999999 0 0
EOS Console [root://localhost] |/eos/> fs status 65
# ------------------------------------------------------------------------------------
# FileSystem Variables
# ....................................................................................
bootcheck := 0
configstatus := drain
headroom := 5000000000
host := cmseos5.fnal.gov
hostport := cmseos5.fnal.gov:1095
id := 65
path := /storage/data1
port := 1095
queue := /eos/cmseos5.fnal.gov:1095/fst
queuepath := /eos/cmseos5.fnal.gov:1095/fst/storage/data1
scaninterval := 2419200
schedgroup := default.0
stat.active := online
stat.balancer.running := 0
stat.boot := booted
stat.disk.bw := 144.000000
stat.disk.iops := 69
stat.disk.load := 0.000000
stat.disk.readratemb := 0.000000
stat.disk.writeratemb := 0.001160
stat.drain := stalling
stat.drainbytesleft := 284262400
stat.drainer := on
stat.drainer.running := 0
stat.drainfiles := 190
stat.drainprogress := 0
stat.drainretry := 0
stat.fsck.d_cx_diff := 1
stat.fsck.d_mem_sz_diff := 1
stat.fsck.d_sync_n := 8071
stat.fsck.m_cx_diff := 0
stat.fsck.m_mem_sz_diff := 0
stat.fsck.m_sync_n := 8011
stat.fsck.mem_n := 8071
stat.fsck.orphans_n := 0
stat.fsck.rep_diff_n := 1
stat.fsck.rep_missing_n := 0
stat.fsck.unreg_n := 1
stat.geotag :=
stat.heartbeattime := 1449082130
stat.net.ethratemib := 238.000000
stat.net.inratemib := 0.001370
stat.net.outratemib := 0.001550
stat.nominal.filled := 0.000000
stat.ropen := 0
stat.ropen.hotfiles :=
stat.statfs.bavail := 2929988712
stat.statfs.bfree := 2929988712
stat.statfs.blocks := 2930058112
stat.statfs.bsize := 4096
stat.statfs.capacity := 12001518026752
stat.statfs.ffree := 11720342300
stat.statfs.files := 11720363520
stat.statfs.filled := 0.002369
stat.statfs.freebytes := 12001233764352
stat.statfs.fused := 86917120
stat.statfs.namelen := 255
stat.statfs.type := 1481003842
stat.statfs.usedbytes := 284262400
stat.timeleft := 99999999999
stat.usedfiles := 8071
stat.wopen := 0
stat.wopen.hotfiles :=
uuid := e5fb4e25-0ca8-49dc-b5c0-578ea761637b
# ....................................................................................
# Risk Analysis
# ....................................................................................
number of files := 190 (100.00%)
files healthy := 0 (0.00%)
files at risk := 102 (53.68%)
files inaccessbile := 88 (46.32%)
files pending deletion := 0
# ------------------------------------------------------------------------------------
EOS Console [root://localhost] |/eos/> fs dumpmd 65 -path
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_20/cache/nll_der-toys:0.0-wp2000-08800fe638.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_5/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_27/nll_der-toys:0.033343-wp2500-fc6f053826.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_21/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_18/plots/limit_band_plot-log-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_7/nll_der-toys:0.0-wp2000-2cf1fa674e.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_17/cache/nll_der-toys:0.0-wp3000-709e191ee4.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_18/cache/nll_der-toys:0.0-wp3000-6ad6ecf85a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_19/cache/nll_der-toys:0.0-wp2500-492e65cad2.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_17/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_22/nll_der-toys:0.0-wp2500-3aeb8251df.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_23/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_4/cache/nll_der-toys:0.0-wp2500-77680253b0.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_16/nll_der-toys:0.0-wp2500-9b3e4fe432.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_12/cache/quant-toys:0-wp2000-55ee068e8d.db
path=/lost+found/orphans/2261676/c7592d32-59ad-11e5-bbc8-001e67abf518-158-0-logArchive.tar.gz.125305362
path=/lost+found/orphans/2260531/31eddb48-59b3-11e5-bbc8-001e67abf518-60-0-logArchive.tar.gz.125175327
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_2/cache/nll_der-toys:0.0-wp2500-892cfe55de.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_4.py
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_12/model_summary_general.thtml
path=/lost+found/orphans/2261093/4e61074a-59dc-11e5-adda-a0369f23d01e-942-0-logArchive.tar.gz.125222436
path=/lost+found/orphans/2261093/4e61074a-59dc-11e5-adda-a0369f23d01e-957-0-logArchive.tar.gz.125231140
path=/lost+found/orphans/2261117/4e61074a-59dc-11e5-adda-a0369f23d01e-132-0-logArchive.tar.gz.125247526
path=/lost+found/orphans/2266398/752830fe-5b87-11e5-9b56-02163e00d7b3-160-0-logArchive.tar.gz.125677607
path=/lost+found/orphans/2260548/31eddb48-59b3-11e5-bbc8-001e67abf518-869-0-logArchive.tar.gz.125166122
path=/lost+found/orphans/2260219/55d5b21c-59d9-11e5-a196-a0369f23d050-27-0-logArchive.tar.gz.125176877
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_2/quant-toys:0-wp2000-e0de1a86cc.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_2/cache/nll_der-toys:0.114590-wp2000-cde86f9252.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5/cache/nll_der-toys:0.0-wp2000-33e19f5499.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_18/model_summary_rates.thtml
path=/lost+found/orphans/2260979/81162ad6-59fe-11e5-a196-a0369f23d050-9-0-logArchive.tar.gz.125210674
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1743-0-logArchive.tar.gz.125236285
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_2/cache/nll_der-toys:0.114590-wp2000-ca9b54a228.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_23/quant-toys:0-wp3000-c0fdb49949.cfg
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1310-0-logArchive.tar.gz.125223488
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_8/quant-toys:0-wp2000-1603ad9e10.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_19/cache/nll_der-toys:0.0-wp3000-52f265da48.cfg
path=/lost+found/orphans/2266398/1fd19ff6-5b86-11e5-9b56-02163e00d7b3-182-0-logArchive.tar.gz.125659209
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_11/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_25/cache/nll_der-toys:0.010756-wp3000-654bfc0c28.cfg
path=/lost+found/orphans/2260979/d37b909c-5a01-11e5-a196-a0369f23d050-9-0-logArchive.tar.gz.125213267
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_7/quant-toys:0-wp2000-078ebde551.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_8/nll_der-toys:0.0-wp3000-3d4bd1084b.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_26/cache/nll_der-toys:0.010756-wp3000-e24971f6e6.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_18/nll_der-toys:0.0-wp2000-2c1044e6e9.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_22/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/muon_ge1btags_IDMLJet350Lep150MET100_discovery.json
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_24/cache/nll_der-toys:0.114590-wp2000-09bb227562.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_23/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_1/cache/nll_der-toys:0.0-wp2000-c37adecc24.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_1/cache/nll_der-toys:0.010756-wp3000-7f46bd0c74.db
path=/lost+found/orphans/2260658/6c80894a-59c2-11e5-bbc8-001e67abf518-703-0-logArchive.tar.gz.125187177
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_14/nll_der-toys:0.010756-wp3000-ede352e3bb.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_23/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_12.stdout
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_19/nll_der-toys:0.0-wp2000-c47123d2ad.cfg
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015C-PromptReco-v1.DoubleMuon/jobExecCondor_Run2015C-PromptReco-v1.DoubleMuon_160_RA2AnalysisTree_exp_302723.stdout
path=/eos/uscms/store/user/ptan/work/sl6/development/CMSSW_7_1_20/src/run_0002/work_10000_3934/powhegbox_Z/pwhg_checklimits
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_12/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_7/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150807_204321/0000/B2GEDMNtuple_182.root
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_19/quant-toys:0-wp3000-d7c940dfa2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_17/cache/quant-toys:0-wp3000-5c948e1f9b.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_11/nll_der-toys:0.0-wp3000-2854f18738.cfg
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1241-0-logArchive.tar.gz.125227666
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_13/cache/nll_der-toys:0.0-wp3000-da74ec1b17.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_18/cache/quant-toys:0-wp2500-3e2c957b6a.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_16/quant-toys:0-wp2000-21e186f39d.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_3/nll_der-toys:0.0-wp2500-9bc3e22fe3.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_1/cache/nll_der-toys:0.0-wp2500-dadea2ba5c.db
path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-571-0-logArchive.tar.gz.125182625
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_2/cache/nll_der-toys:0.033343-wp2500-f8444f4c5b.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_19.stdout
path=/lost+found/orphans/2179013/cmsRun_14.log.tar.gz.120396459
path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150807_204321/0000/log/cmsRun_105.log.tar.gz
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1666-0-logArchive.tar.gz.125234346
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_20/cache/nll_der-toys:0.114590-wp2000-d3ff2f68d2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_20/nll_der-toys:0.0-wp2000-742b9015af.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_8/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_11/cache/nll_der-toys:0.0-wp2500-042cc09f4a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_27/cache/nll_der-toys:0.0-wp2000-6b8ec22d15.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_10/nll_der-toys:0.033343-wp2500-ad3ee63d5f.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/quant-toys:0-wp2500-960fc6ef76.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_9/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_12/cache/nll_der-toys:0.0-wp2000-b8d96aa8e2.db
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015C-PromptReco-v1.JetHT/jobExecCondor_Run2015C-PromptReco-v1.JetHT_212_RA2AnalysisTree_exp_303823.stderr
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_143_RA2AnalysisTree_exp_230053.condor
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_24/nll_der-toys:0.033343-wp2500-a38cf86a78.cfg
path=/lost+found/orphans/2260469/89620134-59ad-11e5-bbc8-001e67abf518-1110-0-logArchive.tar.gz.125145807
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/electron_ge1btags_IDTLJet250Lep150MET50_discovery.json
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_26/model_summary_rates.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_24/cache/quant-toys:0-wp2500-9e44a880a5.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_24/nll_der-toys:0.0-wp2000-413b0f39e1.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/quant-toys:0-wp2500-83255dd362.db
path=/lost+found/orphans/2260469/ce14512a-59b1-11e5-bbc8-001e67abf518-248-0-logArchive.tar.gz.125160152
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_25/cache/nll_der-toys:0.033343-wp2500-b968cc1080.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_27/cache/quant-toys:0-wp2500-0890c5e533.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_23/cache/nll_der-toys:0.0-wp2000-7f03519e3a.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_2/nll_der-toys:0.0-wp2500-35b193a804.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_15/quant-toys:0-wp2000-4b5e3c2807.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_22/cache/nll_der-toys:0.0-wp2000-50aedc77a1.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_18/cache/nll_der-toys:0.0-wp3000-ded9d44ab2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_20/nll_der-toys:0.033343-wp2500-7a8fe444c2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/electron_bayesian_limits_expected_ge1btags_IDTTJet250Lep120MET50.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_21/nll_der-toys:0.114590-wp2000-397c5ab425.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/quant-toys:0-wp2500-83255dd362.cfg
path=/lost+found/orphans/2260979/098edc8c-5a04-11e5-a196-a0369f23d050-20-0-logArchive.tar.gz.125214997
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_15/cache/nll_der-toys:0.0-wp2500-830ddf6a3a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_10/cache/quant-toys:0-wp2500-e8b4ae4b04.db
path=/lost+found/orphans/2260979/16da51ee-5a01-11e5-a196-a0369f23d050-24-0-logArchive.tar.gz.125212448
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_2/nll_der-toys:0.010756-wp3000-b9ebd89fdc.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_3/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_25/cache/quant-toys:0-wp3000-9ec7dd6f7d.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_13/cache/nll_der-toys:0.0-wp2000-1eea988edb.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_14/cache/nll_der-toys:0.0-wp3000-dd7af868cb.cfg
path=/lost+found/orphans/2260837/c5571204-59dc-11e5-a196-a0369f23d050-31-0-logArchive.tar.gz.125183286
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_12.condor
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_19/cache/quant-toys:0-wp2500-94ec3f53d8.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_6/quant-toys:0-wp3000-2cdf240713.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_2/plots/elec_invmass_ge1btags_stack.png
path=/lost+found/orphans/2404866/Spring15.TTJets_HT-800to1200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8_Expectation.root.134833998
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.MET/jobExecCondor_Run2015D-PromptReco-v3.MET_23_RA2AnalysisTree_exp_276208.stderr
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_10/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_2/plots/elec_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_45_RA2AnalysisTree_exp_230093.condor
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_18/cache/nll_der-toys:0.033343-wp2500-af448fdb44.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_6/model_summary_rates.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_8.condor
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1618-0-logArchive.tar.gz.125225831
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_14/nll_der-toys:0.0-wp2500-3951cd37ee.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_16/cache/quant-toys:0-wp2500-252ee6b98b.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_6/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_3/nll_der-toys:0.010756-wp3000-ea4c2607fd.cfg
path=/lost+found/orphans/2260598/fa0f2974-59bd-11e5-bbc8-001e67abf518-223-0-logArchive.tar.gz.125187442
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_18/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_5/cache/nll_der-toys:0.114590-wp2000-9663860664.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_7/model_summary_nuisance.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_20/quant-toys:0-wp3000-59ee280091.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/muon_bayesian_limits_expected_ge1btags_IDMTJet350Lep120MET100.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_25/nll_der-toys:0.0-wp2500-b7bb2c1c5f.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_14/cache/nll_der-toys:0.010756-wp3000-da6ecf89d3.cfg
path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150805_210241/0000/failed/log/cmsRun_35.log.tar.gz
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_20/cache/quant-toys:0-wp3000-7d758a0ac8.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_23/nll_der-toys:0.0-wp2500-9541ba1e45.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_19/cache/nll_der-toys:0.114590-wp2000-af28f86990.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_21/cache/quant-toys:0-wp3000-9d95c2212a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/electron_ge1btags_IDTTJet250Lep120MET100_discovery.json
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_25/cache/nll_der-toys:0.033343-wp2500-b78726f0e7.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_18/nll_der-toys:0.033343-wp2500-b8addcf58a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/electron_bayesian_limits_expected_ge1btags_IDTLJet450Lep120MET50.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_4/cache/nll_der-toys:0.0-wp3000-cd4c3112fd.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_14/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_23/nll_der-toys:0.114590-wp2000-129573224a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_17/cache/nll_der-toys:0.0-wp3000-2838ccb834.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_25/quant-toys:0-wp3000-34fb42cbdc.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_18/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_9/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_3/cache/nll_der-toys:0.033343-wp2500-c487654e3e.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3/nll_der-toys:0.010756-wp3000-b856611bdb.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3.condor
path=/lost+found/orphans/2180880/cmsRun_47.log.tar.gz.120497592
path=/lost+found/orphans/2260548/186434d6-59ba-11e5-bbc8-001e67abf518-108-0-logArchive.tar.gz.125177266
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_4/cache/nll_der-toys:0.114590-wp2000-1ec6a96890.cfg
path=/lost+found/orphans/2179013/cmsRun_46.log.tar.gz.120394179
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/electron_bayesian_limits_expected_ge1btags_IDMLJet450Lep150MET150.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_24/cache/quant-toys:0-wp3000-aa781ee72c.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5.condor
path=/eos/uscms/store/user/lpcmbja/noreplica/zzDR_p0plusp1_050915_correlatedBkg_sigThrsld2.5_deCorrdMedHighNuis_trendGuessObserved_snapshot_fix/condor.VVVh_CMS_common_expected_minuit2_0p0005_tries5_strategy1.log
path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-522-0-logArchive.tar.gz.125193169
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_6/cache/nll_der-toys:0.0-wp2000-0ddfeb4e7d.cfg
path=/lost+found/orphans/2260219/3f9c812a-59d3-11e5-a196-a0369f23d050-29-0-logArchive.tar.gz.125171158
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_24/cache/nll_der-toys:0.114590-wp2000-09bb227562.db
path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-286-0-logArchive.tar.gz.125191132
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_14/cache/quant-toys:0-wp2000-fef6a41b4c.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_27.csh
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_17/cache/quant-toys:0-wp2500-78635dd5c4.cfg
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_84_RA2AnalysisTree_exp_230132.stdout
path=/lost+found/orphans/2260469/89620134-59ad-11e5-bbc8-001e67abf518-1241-0-logArchive.tar.gz.125152227
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_8/cache/nll_der-toys:0.0-wp2000-68eb4a697e.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_24/nll_der-toys:0.0-wp2500-185549eb2f.cfg
path=/lost+found/orphans/2260979/a5e483f0-59fc-11e5-a196-a0369f23d050-13-0-logArchive.tar.gz.125209064
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_15/cache/nll_der-toys:0.0-wp2000-7af3e0c80e.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_15/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_17/model_summary_general.thtml
path=/lost+found/orphans/2260548/31eddb48-59b3-11e5-bbc8-001e67abf518-730-0-logArchive.tar.gz.125180918
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/nll_der-toys:0.114590-wp2000-d68574c190.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_14/cache/nll_der-toys:0.0-wp2000-c3e1f98183.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
EOS Console [root://localhost] |/eos/> file info /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
File: '/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg' Flags: 0644
Size: 7205
Modify: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.100891000
Change: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.89492856
CUid: 47452 CGid: 5063 Fxid: 081c93ff Fid: 136090623 Pid: 2434745 Pxid: 002526b9
XStype: adler XS: d1 a1 8a 64 ETAG: 36531548442329088:d1a18a64
replica Stripes: 2 Blocksize: 4k LayoutId: 00100112
#Rep: 2
# fs-id #...................................................................................................................................
# host # schedgroup # path # boot # configstatus # drain # active # geotag
#...................................................................................................................................
0 107 cmseos27.fnal.gov default.0 /storage/data1 booted rw nodrain online
1 65 cmseos5.fnal.gov default.0 /storage/data1 booted drain stalling online
*******
EOS Console [root://localhost] |/eos/> file check /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
path="/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg" fid="081c93ff" size="7205" nrep="2" checksumtype="adler" checksum="d1a18a6400000000000000000000000000000000"
error: unable to retrieve file meta data from cmseos27.fnal.gov:1095 [61]
error: unable to retrieve file meta data from cmseos5.fnal.gov:1095 [61]
EOS Console [root://localhost] |/eos/> file verify /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
success: sending verify to fsid= 107 for path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
success: sending verify to fsid= 65 for path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
EOS Console [root://localhost] |/eos/> file check /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
path="/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg" fid="081c93ff" size="7205" nrep="2" checksumtype="adler" checksum="d1a18a6400000000000000000000000000000000"
error: unable to retrieve file meta data from cmseos27.fnal.gov:1095 [61]
error: unable to retrieve file meta data from cmseos5.fnal.gov:1095 [61]
EOS Console [root://localhost] |/eos/> file info /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
File: '/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg' Flags: 0644
Size: 7205
Modify: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.100891000
Change: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.89492856
CUid: 47452 CGid: 5063 Fxid: 081c93ff Fid: 136090623 Pid: 2434745 Pxid: 002526b9
XStype: adler XS: d1 a1 8a 64 ETAG: 36531548442329088:d1a18a64
replica Stripes: 2 Blocksize: 4k LayoutId: 00100112
#Rep: 2
# fs-id #...................................................................................................................................
# host # schedgroup # path # boot # configstatus # drain # active # geotag
#...................................................................................................................................
0 107 cmseos27.fnal.gov default.0 /storage/data1 booted rw nodrain online
1 65 cmseos5.fnal.gov default.0 /storage/data1 booted drain stalling online
*******
EOS Console [root://localhost] |/eos/>
This basically says this file does NOT exist on the FSTs, on the FST I see consistent errors:
151202 12:50:44 8029 XrootdXeq: root.27203:82@cmssrv222 login as daemon 151202 12:50:44 8029 FstOfs_stat: root.27203:82@cmssrv222 Unable to stat file /storage/data1/00003529/081c93ff; no such file or directory 151202 12:50:44 time=1449082244.465897 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17bebc3700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:50:44 time=1449082244.465941 func=FSctl level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17bebc3700 source=XrdFstOfs:1297 tident= sec=(null) uid=0 gid=0 name=- geo="" no fmd for fileid 136090623 on filesystem 65 151202 12:51:07 time=1449082267.380189 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:51:07 time=1449082267.380255 func=Verify level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=Verify:108 tident= sec=(null) uid=0 gid=0 name=- geo="" unable to verify file id=81c93ff on fs=65 path=/storage/data1/00003529/081c93ff - stat on local disk failed 151202 12:51:07 time=1449082267.380282 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:51:07 time=1449082267.380305 func=Verify level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=Verify:122 tident= sec=(null) uid=0 gid=0 name=- geo="" unable to verify id=81c93ff on fs=65 path=/storage/data1/00003529/081c93ff - no local MD stored 151202 12:51:25 11252 FstOfs_stat: root.27203:82@cmssrv222 Unable to stat file /storage/data1/00003529/081c93ff; no such file or directory 151202 12:51:25 time=1449082285.467642 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f171b4d3700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:51:25 time=1449082285.467693 func=FSctl level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f171b4d3700 source=XrdFstOfs:1297 tident= sec=(null) uid=0 gid=0 name=- geo="" no fmd for fileid 136090623 on filesystem 65
#4 Updated by Gerard Bernabeu Altayo about 4 years ago
- Status changed from New to Resolved
all this nodes have been retired now.