Bug #10763
Retire old EOS nodes (cmseos2-5)
0%
Description
The following nodes should be retired: cmseos2,3,4,5. This are all 2x1Gbps nodes and very old.
I got the following email from David Fagan:
These are old
cmseos2,3,4,5 (disk 2009) 36TB
?
cmseos1 (2011) 108TB
-----
Getting there...
cmseos27 (2012) 162TB
For now I'd keep cmseos1 and 27, they both have 10GE NICs.
I think I'll need to write a small procedure on how to migrate disks.
Subtasks
History
#1 Updated by Gerard Bernabeu Altayo over 5 years ago
Drain documentation available at http://eos.readthedocs.org/en/latest/configuration/draining.html
It does not make sense to make our own parallel documentation, I will just post in this ticket what I've done exactly for future reference:
[root@cmssrv222 ~]# eos fs ls -d #.................................................................................................................................... # host (#...) # id # path # drain # progress # files # bytes-left # timeleft #retry #wopen #.................................................................................................................................... [root@cmssrv222 ~]# eos fs ls | grep cmseos2 cmseos2.fnal.gov (1095) 74 /storage/data1 default.0 booted rw nodrain online cmseos2.fnal.gov (1095) 75 /storage/data2 default.1 booted rw nodrain online cmseos2.fnal.gov (1095) 76 /storage/data3 default.3 booted rw nodrain online cmseos25.fnal.gov (1095) 103 /storage/data1 default.0 booted rw nodrain online cmseos25.fnal.gov (1095) 104 /storage/data2 default.1 booted rw nodrain online cmseos26.fnal.gov (1095) 105 /storage/data1 default.2 booted rw nodrain online cmseos26.fnal.gov (1095) 106 /storage/data2 default.3 booted rw nodrain online cmseos27.fnal.gov (1095) 107 /storage/data1 default.0 booted rw nodrain online cmseos27.fnal.gov (1095) 108 /storage/data2 default.1 booted rw nodrain online cmseos27.fnal.gov (1095) 109 /storage/data3 default.2 booted rw nodrain online cmseos28.fnal.gov (1095) 110 /storage/data1 default.0 booted rw nodrain online cmseos28.fnal.gov (1095) 111 /storage/data2 default.1 booted rw nodrain online cmseos29.fnal.gov (1095) 112 /storage/data1 default.2 booted rw nodrain online cmseos29.fnal.gov (1095) 113 /storage/data2 default.3 booted rw nodrain online [root@cmssrv222 ~]# eos fs config 74 configstatus=drain [root@cmssrv222 ~]# eos fs config 75 configstatus=drain [root@cmssrv222 ~]# eos fs config 76 configstatus=drain [root@cmssrv222 ~]# eos fs ls | grep cmseos3.fnal cmseos3.fnal.gov (1095) 71 /storage/data1 default.0 booted rw nodrain online cmseos3.fnal.gov (1095) 72 /storage/data2 default.2 booted rw nodrain online cmseos3.fnal.gov (1095) 73 /storage/data3 default.1 booted rw nodrain online [root@cmssrv222 ~]# eos fs config 71 configstatus=drain [root@cmssrv222 ~]# eos fs config 72 configstatus=drain [root@cmssrv222 ~]# eos fs config 73 configstatus=drain [root@cmssrv222 ~]# eos fs ls | grep cmseos4.fnal cmseos4.fnal.gov (1095) 68 /storage/data1 default.0 booted rw nodrain online cmseos4.fnal.gov (1095) 69 /storage/data2 default.1 booted rw nodrain online cmseos4.fnal.gov (1095) 70 /storage/data3 default.2 booted rw nodrain online [root@cmssrv222 ~]# eos fs config 68 configstatus=drain [root@cmssrv222 ~]# eos fs config 69 configstatus=drain [root@cmssrv222 ~]# eos fs config 70 configstatus=drain [root@cmssrv222 ~]# eos fs ls | grep cmseos5.fnal cmseos5.fnal.gov (1095) 65 /storage/data1 default.0 booted rw nodrain online cmseos5.fnal.gov (1095) 66 /storage/data2 default.1 booted rw nodrain online cmseos5.fnal.gov (1095) 67 /storage/data3 default.2 booted rw nodrain online [root@cmssrv222 ~]# eos fs config 65 configstatus=drain [root@cmssrv222 ~]# eos fs config 66 configstatus=drain [root@cmssrv222 ~]# eos fs config 67 configstatus=drain [root@cmssrv222 ~]# eos fs ls -d #.................................................................................................................................... # host (#...) # id # path # drain # progress # files # bytes-left # timeleft #retry #wopen #.................................................................................................................................... cmseos5.fnal.gov (1095) 65 /storage/data1 prepare 0 0.00 0.00 B 50 0 4 cmseos5.fnal.gov (1095) 66 /storage/data2 prepare 0 0.00 0.00 B 53 0 0 cmseos5.fnal.gov (1095) 67 /storage/data3 prepare 0 0.00 0.00 B 57 0 4 cmseos4.fnal.gov (1095) 68 /storage/data1 prepare 0 0.00 0.00 B 28 0 3 cmseos4.fnal.gov (1095) 69 /storage/data2 prepare 0 0.00 0.00 B 32 0 1 cmseos4.fnal.gov (1095) 70 /storage/data3 prepare 0 0.00 0.00 B 36 0 2 cmseos3.fnal.gov (1095) 71 /storage/data1 prepare 0 0.00 0.00 B 3 0 5 cmseos3.fnal.gov (1095) 72 /storage/data2 prepare 0 0.00 0.00 B 7 0 0 cmseos3.fnal.gov (1095) 73 /storage/data3 prepare 0 0.00 0.00 B 10 0 2 cmseos2.fnal.gov (1095) 74 /storage/data1 draining 0 143.51 k 7.59 TB 86298 0 0 cmseos2.fnal.gov (1095) 75 /storage/data2 draining 0 138.33 k 7.53 TB 86324 0 2 cmseos2.fnal.gov (1095) 76 /storage/data3 draining 0 152.90 k 8.09 TB 86328 0 2 [root@cmssrv222 ~]#
#2 Updated by Gerard Bernabeu Altayo over 5 years ago
- Subject changed from Retire old EOS nodes to Retire old EOS nodes (cmseos2-5)
The drain expired for some nodes, resubmitted it with:
[root@cmssrv222 ~]# for i in `seq 65 76`; do eos fs config $i configstatus=drain; done
#3 Updated by Gerard Bernabeu Altayo about 5 years ago
All nodes have some issues with files, focusing on cmseos5 first I think that there is inevitable data loss on it:
EOS Console [root://localhost] |/eos/> fs ls -d #.................................................................................................................................... # host (#...) # id # path # drain # progress # files # bytes-left # timeleft #retry #wopen #.................................................................................................................................... cmseos5.fnal.gov (1095) 65 /storage/data1 stalling 0 190.00 284.26 MB 99999999999 0 0 cmseos5.fnal.gov (1095) 66 /storage/data2 stalling 0 251.00 3.50 GB 99999999999 0 0 cmseos5.fnal.gov (1095) 67 /storage/data3 stalling 0 240.00 2.84 GB 99999999999 0 0 EOS Console [root://localhost] |/eos/> fs status 65 # ------------------------------------------------------------------------------------ # FileSystem Variables # .................................................................................... bootcheck := 0 configstatus := drain headroom := 5000000000 host := cmseos5.fnal.gov hostport := cmseos5.fnal.gov:1095 id := 65 path := /storage/data1 port := 1095 queue := /eos/cmseos5.fnal.gov:1095/fst queuepath := /eos/cmseos5.fnal.gov:1095/fst/storage/data1 scaninterval := 2419200 schedgroup := default.0 stat.active := online stat.balancer.running := 0 stat.boot := booted stat.disk.bw := 144.000000 stat.disk.iops := 69 stat.disk.load := 0.000000 stat.disk.readratemb := 0.000000 stat.disk.writeratemb := 0.001160 stat.drain := stalling stat.drainbytesleft := 284262400 stat.drainer := on stat.drainer.running := 0 stat.drainfiles := 190 stat.drainprogress := 0 stat.drainretry := 0 stat.fsck.d_cx_diff := 1 stat.fsck.d_mem_sz_diff := 1 stat.fsck.d_sync_n := 8071 stat.fsck.m_cx_diff := 0 stat.fsck.m_mem_sz_diff := 0 stat.fsck.m_sync_n := 8011 stat.fsck.mem_n := 8071 stat.fsck.orphans_n := 0 stat.fsck.rep_diff_n := 1 stat.fsck.rep_missing_n := 0 stat.fsck.unreg_n := 1 stat.geotag := stat.heartbeattime := 1449082130 stat.net.ethratemib := 238.000000 stat.net.inratemib := 0.001370 stat.net.outratemib := 0.001550 stat.nominal.filled := 0.000000 stat.ropen := 0 stat.ropen.hotfiles := stat.statfs.bavail := 2929988712 stat.statfs.bfree := 2929988712 stat.statfs.blocks := 2930058112 stat.statfs.bsize := 4096 stat.statfs.capacity := 12001518026752 stat.statfs.ffree := 11720342300 stat.statfs.files := 11720363520 stat.statfs.filled := 0.002369 stat.statfs.freebytes := 12001233764352 stat.statfs.fused := 86917120 stat.statfs.namelen := 255 stat.statfs.type := 1481003842 stat.statfs.usedbytes := 284262400 stat.timeleft := 99999999999 stat.usedfiles := 8071 stat.wopen := 0 stat.wopen.hotfiles := uuid := e5fb4e25-0ca8-49dc-b5c0-578ea761637b # .................................................................................... # Risk Analysis # .................................................................................... number of files := 190 (100.00%) files healthy := 0 (0.00%) files at risk := 102 (53.68%) files inaccessbile := 88 (46.32%) files pending deletion := 0 # ------------------------------------------------------------------------------------ EOS Console [root://localhost] |/eos/> fs dumpmd 65 -path path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_20/cache/nll_der-toys:0.0-wp2000-08800fe638.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_5/plots/mu_invmass_ge1btags_stack.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_27/nll_der-toys:0.033343-wp2500-fc6f053826.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_21/model_plots.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_18/plots/limit_band_plot-log-bayesian.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_7/nll_der-toys:0.0-wp2000-2cf1fa674e.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_17/cache/nll_der-toys:0.0-wp3000-709e191ee4.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_18/cache/nll_der-toys:0.0-wp3000-6ad6ecf85a.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_19/cache/nll_der-toys:0.0-wp2500-492e65cad2.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_17/index.html path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_22/nll_der-toys:0.0-wp2500-3aeb8251df.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_23/model_plots.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_4/cache/nll_der-toys:0.0-wp2500-77680253b0.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_16/nll_der-toys:0.0-wp2500-9b3e4fe432.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_12/cache/quant-toys:0-wp2000-55ee068e8d.db path=/lost+found/orphans/2261676/c7592d32-59ad-11e5-bbc8-001e67abf518-158-0-logArchive.tar.gz.125305362 path=/lost+found/orphans/2260531/31eddb48-59b3-11e5-bbc8-001e67abf518-60-0-logArchive.tar.gz.125175327 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_2/cache/nll_der-toys:0.0-wp2500-892cfe55de.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_4.py path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_12/model_summary_general.thtml path=/lost+found/orphans/2261093/4e61074a-59dc-11e5-adda-a0369f23d01e-942-0-logArchive.tar.gz.125222436 path=/lost+found/orphans/2261093/4e61074a-59dc-11e5-adda-a0369f23d01e-957-0-logArchive.tar.gz.125231140 path=/lost+found/orphans/2261117/4e61074a-59dc-11e5-adda-a0369f23d01e-132-0-logArchive.tar.gz.125247526 path=/lost+found/orphans/2266398/752830fe-5b87-11e5-9b56-02163e00d7b3-160-0-logArchive.tar.gz.125677607 path=/lost+found/orphans/2260548/31eddb48-59b3-11e5-bbc8-001e67abf518-869-0-logArchive.tar.gz.125166122 path=/lost+found/orphans/2260219/55d5b21c-59d9-11e5-a196-a0369f23d050-27-0-logArchive.tar.gz.125176877 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_2/quant-toys:0-wp2000-e0de1a86cc.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_2/cache/nll_der-toys:0.114590-wp2000-cde86f9252.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5/cache/nll_der-toys:0.0-wp2000-33e19f5499.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_18/model_summary_rates.thtml path=/lost+found/orphans/2260979/81162ad6-59fe-11e5-a196-a0369f23d050-9-0-logArchive.tar.gz.125210674 path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1743-0-logArchive.tar.gz.125236285 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_2/cache/nll_der-toys:0.114590-wp2000-ca9b54a228.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_23/quant-toys:0-wp3000-c0fdb49949.cfg path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1310-0-logArchive.tar.gz.125223488 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_8/quant-toys:0-wp2000-1603ad9e10.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_19/cache/nll_der-toys:0.0-wp3000-52f265da48.cfg path=/lost+found/orphans/2266398/1fd19ff6-5b86-11e5-9b56-02163e00d7b3-182-0-logArchive.tar.gz.125659209 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_11/model_summary_rate_impacts.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_25/cache/nll_der-toys:0.010756-wp3000-654bfc0c28.cfg path=/lost+found/orphans/2260979/d37b909c-5a01-11e5-a196-a0369f23d050-9-0-logArchive.tar.gz.125213267 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_7/quant-toys:0-wp2000-078ebde551.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_8/nll_der-toys:0.0-wp3000-3d4bd1084b.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_26/cache/nll_der-toys:0.010756-wp3000-e24971f6e6.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_18/nll_der-toys:0.0-wp2000-2c1044e6e9.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_22/plots/limit_band_plot-bayesian.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/muon_ge1btags_IDMLJet350Lep150MET100_discovery.json path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_24/cache/nll_der-toys:0.114590-wp2000-09bb227562.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_23/plots/limit_band_plot-bayesian.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_1/cache/nll_der-toys:0.0-wp2000-c37adecc24.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_1/cache/nll_der-toys:0.010756-wp3000-7f46bd0c74.db path=/lost+found/orphans/2260658/6c80894a-59c2-11e5-bbc8-001e67abf518-703-0-logArchive.tar.gz.125187177 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_14/nll_der-toys:0.010756-wp3000-ede352e3bb.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_23/index.html path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_12.stdout path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_19/nll_der-toys:0.0-wp2000-c47123d2ad.cfg path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015C-PromptReco-v1.DoubleMuon/jobExecCondor_Run2015C-PromptReco-v1.DoubleMuon_160_RA2AnalysisTree_exp_302723.stdout path=/eos/uscms/store/user/ptan/work/sl6/development/CMSSW_7_1_20/src/run_0002/work_10000_3934/powhegbox_Z/pwhg_checklimits path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_12/plots/limit_band_plot-bayesian.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_7/model_summary_rate_impacts.thtml path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150807_204321/0000/B2GEDMNtuple_182.root path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_19/quant-toys:0-wp3000-d7c940dfa2.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_17/cache/quant-toys:0-wp3000-5c948e1f9b.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_11/nll_der-toys:0.0-wp3000-2854f18738.cfg path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1241-0-logArchive.tar.gz.125227666 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_13/cache/nll_der-toys:0.0-wp3000-da74ec1b17.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_18/cache/quant-toys:0-wp2500-3e2c957b6a.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_16/quant-toys:0-wp2000-21e186f39d.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_3/nll_der-toys:0.0-wp2500-9bc3e22fe3.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_1/cache/nll_der-toys:0.0-wp2500-dadea2ba5c.db path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-571-0-logArchive.tar.gz.125182625 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_2/cache/nll_der-toys:0.033343-wp2500-f8444f4c5b.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_19.stdout path=/lost+found/orphans/2179013/cmsRun_14.log.tar.gz.120396459 path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150807_204321/0000/log/cmsRun_105.log.tar.gz path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1666-0-logArchive.tar.gz.125234346 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_20/cache/nll_der-toys:0.114590-wp2000-d3ff2f68d2.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_20/nll_der-toys:0.0-wp2000-742b9015af.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_8/model_summary_rate_impacts.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_11/cache/nll_der-toys:0.0-wp2500-042cc09f4a.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_27/cache/nll_der-toys:0.0-wp2000-6b8ec22d15.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_10/nll_der-toys:0.033343-wp2500-ad3ee63d5f.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/quant-toys:0-wp2500-960fc6ef76.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_9/index.html path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_12/cache/nll_der-toys:0.0-wp2000-b8d96aa8e2.db path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015C-PromptReco-v1.JetHT/jobExecCondor_Run2015C-PromptReco-v1.JetHT_212_RA2AnalysisTree_exp_303823.stderr path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_143_RA2AnalysisTree_exp_230053.condor path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_24/nll_der-toys:0.033343-wp2500-a38cf86a78.cfg path=/lost+found/orphans/2260469/89620134-59ad-11e5-bbc8-001e67abf518-1110-0-logArchive.tar.gz.125145807 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/electron_ge1btags_IDTLJet250Lep150MET50_discovery.json path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_26/model_summary_rates.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_24/cache/quant-toys:0-wp2500-9e44a880a5.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_24/nll_der-toys:0.0-wp2000-413b0f39e1.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/quant-toys:0-wp2500-83255dd362.db path=/lost+found/orphans/2260469/ce14512a-59b1-11e5-bbc8-001e67abf518-248-0-logArchive.tar.gz.125160152 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_25/cache/nll_der-toys:0.033343-wp2500-b968cc1080.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_27/cache/quant-toys:0-wp2500-0890c5e533.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_23/cache/nll_der-toys:0.0-wp2000-7f03519e3a.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_2/nll_der-toys:0.0-wp2500-35b193a804.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_15/quant-toys:0-wp2000-4b5e3c2807.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_22/cache/nll_der-toys:0.0-wp2000-50aedc77a1.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_18/cache/nll_der-toys:0.0-wp3000-ded9d44ab2.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_20/nll_der-toys:0.033343-wp2500-7a8fe444c2.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/electron_bayesian_limits_expected_ge1btags_IDTTJet250Lep120MET50.txt path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_21/nll_der-toys:0.114590-wp2000-397c5ab425.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/quant-toys:0-wp2500-83255dd362.cfg path=/lost+found/orphans/2260979/098edc8c-5a04-11e5-a196-a0369f23d050-20-0-logArchive.tar.gz.125214997 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_15/cache/nll_der-toys:0.0-wp2500-830ddf6a3a.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_10/cache/quant-toys:0-wp2500-e8b4ae4b04.db path=/lost+found/orphans/2260979/16da51ee-5a01-11e5-a196-a0369f23d050-24-0-logArchive.tar.gz.125212448 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_2/nll_der-toys:0.010756-wp3000-b9ebd89fdc.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_3/plots/mu_invmass_ge1btags_stack.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_25/cache/quant-toys:0-wp3000-9ec7dd6f7d.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_13/cache/nll_der-toys:0.0-wp2000-1eea988edb.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_14/cache/nll_der-toys:0.0-wp3000-dd7af868cb.cfg path=/lost+found/orphans/2260837/c5571204-59dc-11e5-a196-a0369f23d050-31-0-logArchive.tar.gz.125183286 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_12.condor path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_19/cache/quant-toys:0-wp2500-94ec3f53d8.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_6/quant-toys:0-wp3000-2cdf240713.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_2/plots/elec_invmass_ge1btags_stack.png path=/lost+found/orphans/2404866/Spring15.TTJets_HT-800to1200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8_Expectation.root.134833998 path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.MET/jobExecCondor_Run2015D-PromptReco-v3.MET_23_RA2AnalysisTree_exp_276208.stderr path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_10/plots/limit_band_plot-bayesian.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_2/plots/elec_invmass_ge1btags_stack.png path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_45_RA2AnalysisTree_exp_230093.condor path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_18/cache/nll_der-toys:0.033343-wp2500-af448fdb44.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_6/model_summary_rates.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_8.condor path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1618-0-logArchive.tar.gz.125225831 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3/plots/mu_invmass_ge1btags_stack.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_14/nll_der-toys:0.0-wp2500-3951cd37ee.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_16/cache/quant-toys:0-wp2500-252ee6b98b.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5/plots/mu_invmass_ge1btags_stack.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_6/model_summary_rate_impacts.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_3/nll_der-toys:0.010756-wp3000-ea4c2607fd.cfg path=/lost+found/orphans/2260598/fa0f2974-59bd-11e5-bbc8-001e67abf518-223-0-logArchive.tar.gz.125187442 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_18/index.html path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_5/cache/nll_der-toys:0.114590-wp2000-9663860664.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_7/model_summary_nuisance.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_20/quant-toys:0-wp3000-59ee280091.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/muon_bayesian_limits_expected_ge1btags_IDMTJet350Lep120MET100.txt path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_25/nll_der-toys:0.0-wp2500-b7bb2c1c5f.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_14/cache/nll_der-toys:0.010756-wp3000-da6ecf89d3.cfg path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150805_210241/0000/failed/log/cmsRun_35.log.tar.gz path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_20/cache/quant-toys:0-wp3000-7d758a0ac8.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_23/nll_der-toys:0.0-wp2500-9541ba1e45.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_19/cache/nll_der-toys:0.114590-wp2000-af28f86990.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_21/cache/quant-toys:0-wp3000-9d95c2212a.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/electron_ge1btags_IDTTJet250Lep120MET100_discovery.json path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_25/cache/nll_der-toys:0.033343-wp2500-b78726f0e7.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_18/nll_der-toys:0.033343-wp2500-b8addcf58a.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/electron_bayesian_limits_expected_ge1btags_IDTLJet450Lep120MET50.txt path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_4/cache/nll_der-toys:0.0-wp3000-cd4c3112fd.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_14/plots/mu_invmass_ge1btags_stack.png path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_23/nll_der-toys:0.114590-wp2000-129573224a.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_17/cache/nll_der-toys:0.0-wp3000-2838ccb834.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_25/quant-toys:0-wp3000-34fb42cbdc.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_18/model_plots.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_9/model_plots.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_3/cache/nll_der-toys:0.033343-wp2500-c487654e3e.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3/nll_der-toys:0.010756-wp3000-b856611bdb.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3.condor path=/lost+found/orphans/2180880/cmsRun_47.log.tar.gz.120497592 path=/lost+found/orphans/2260548/186434d6-59ba-11e5-bbc8-001e67abf518-108-0-logArchive.tar.gz.125177266 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_4/cache/nll_der-toys:0.114590-wp2000-1ec6a96890.cfg path=/lost+found/orphans/2179013/cmsRun_46.log.tar.gz.120394179 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/electron_bayesian_limits_expected_ge1btags_IDMLJet450Lep150MET150.txt path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_24/cache/quant-toys:0-wp3000-aa781ee72c.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5.condor path=/eos/uscms/store/user/lpcmbja/noreplica/zzDR_p0plusp1_050915_correlatedBkg_sigThrsld2.5_deCorrdMedHighNuis_trendGuessObserved_snapshot_fix/condor.VVVh_CMS_common_expected_minuit2_0p0005_tries5_strategy1.log path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-522-0-logArchive.tar.gz.125193169 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_6/cache/nll_der-toys:0.0-wp2000-0ddfeb4e7d.cfg path=/lost+found/orphans/2260219/3f9c812a-59d3-11e5-a196-a0369f23d050-29-0-logArchive.tar.gz.125171158 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_24/cache/nll_der-toys:0.114590-wp2000-09bb227562.db path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-286-0-logArchive.tar.gz.125191132 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_14/cache/quant-toys:0-wp2000-fef6a41b4c.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_27.csh path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_17/cache/quant-toys:0-wp2500-78635dd5c4.cfg path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_84_RA2AnalysisTree_exp_230132.stdout path=/lost+found/orphans/2260469/89620134-59ad-11e5-bbc8-001e67abf518-1241-0-logArchive.tar.gz.125152227 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_8/cache/nll_der-toys:0.0-wp2000-68eb4a697e.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_24/nll_der-toys:0.0-wp2500-185549eb2f.cfg path=/lost+found/orphans/2260979/a5e483f0-59fc-11e5-a196-a0369f23d050-13-0-logArchive.tar.gz.125209064 path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_15/cache/nll_der-toys:0.0-wp2000-7af3e0c80e.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_15/model_summary_rate_impacts.thtml path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_17/model_summary_general.thtml path=/lost+found/orphans/2260548/31eddb48-59b3-11e5-bbc8-001e67abf518-730-0-logArchive.tar.gz.125180918 path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/nll_der-toys:0.114590-wp2000-d68574c190.cfg path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_14/cache/nll_der-toys:0.0-wp2000-c3e1f98183.db path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg EOS Console [root://localhost] |/eos/> file info /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg File: '/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg' Flags: 0644 Size: 7205 Modify: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.100891000 Change: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.89492856 CUid: 47452 CGid: 5063 Fxid: 081c93ff Fid: 136090623 Pid: 2434745 Pxid: 002526b9 XStype: adler XS: d1 a1 8a 64 ETAG: 36531548442329088:d1a18a64 replica Stripes: 2 Blocksize: 4k LayoutId: 00100112 #Rep: 2 # fs-id #................................................................................................................................... # host # schedgroup # path # boot # configstatus # drain # active # geotag #................................................................................................................................... 0 107 cmseos27.fnal.gov default.0 /storage/data1 booted rw nodrain online 1 65 cmseos5.fnal.gov default.0 /storage/data1 booted drain stalling online ******* EOS Console [root://localhost] |/eos/> file check /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg path="/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg" fid="081c93ff" size="7205" nrep="2" checksumtype="adler" checksum="d1a18a6400000000000000000000000000000000" error: unable to retrieve file meta data from cmseos27.fnal.gov:1095 [61] error: unable to retrieve file meta data from cmseos5.fnal.gov:1095 [61] EOS Console [root://localhost] |/eos/> file verify /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg success: sending verify to fsid= 107 for path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg success: sending verify to fsid= 65 for path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg EOS Console [root://localhost] |/eos/> file check /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg path="/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg" fid="081c93ff" size="7205" nrep="2" checksumtype="adler" checksum="d1a18a6400000000000000000000000000000000" error: unable to retrieve file meta data from cmseos27.fnal.gov:1095 [61] error: unable to retrieve file meta data from cmseos5.fnal.gov:1095 [61] EOS Console [root://localhost] |/eos/> file info /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg File: '/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg' Flags: 0644 Size: 7205 Modify: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.100891000 Change: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.89492856 CUid: 47452 CGid: 5063 Fxid: 081c93ff Fid: 136090623 Pid: 2434745 Pxid: 002526b9 XStype: adler XS: d1 a1 8a 64 ETAG: 36531548442329088:d1a18a64 replica Stripes: 2 Blocksize: 4k LayoutId: 00100112 #Rep: 2 # fs-id #................................................................................................................................... # host # schedgroup # path # boot # configstatus # drain # active # geotag #................................................................................................................................... 0 107 cmseos27.fnal.gov default.0 /storage/data1 booted rw nodrain online 1 65 cmseos5.fnal.gov default.0 /storage/data1 booted drain stalling online ******* EOS Console [root://localhost] |/eos/>
This basically says this file does NOT exist on the FSTs, on the FST I see consistent errors:
151202 12:50:44 8029 XrootdXeq: root.27203:82@cmssrv222 login as daemon 151202 12:50:44 8029 FstOfs_stat: root.27203:82@cmssrv222 Unable to stat file /storage/data1/00003529/081c93ff; no such file or directory 151202 12:50:44 time=1449082244.465897 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17bebc3700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:50:44 time=1449082244.465941 func=FSctl level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17bebc3700 source=XrdFstOfs:1297 tident= sec=(null) uid=0 gid=0 name=- geo="" no fmd for fileid 136090623 on filesystem 65 151202 12:51:07 time=1449082267.380189 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:51:07 time=1449082267.380255 func=Verify level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=Verify:108 tident= sec=(null) uid=0 gid=0 name=- geo="" unable to verify file id=81c93ff on fs=65 path=/storage/data1/00003529/081c93ff - stat on local disk failed 151202 12:51:07 time=1449082267.380282 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:51:07 time=1449082267.380305 func=Verify level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=Verify:122 tident= sec=(null) uid=0 gid=0 name=- geo="" unable to verify id=81c93ff on fs=65 path=/storage/data1/00003529/081c93ff - no local MD stored 151202 12:51:25 11252 FstOfs_stat: root.27203:82@cmssrv222 Unable to stat file /storage/data1/00003529/081c93ff; no such file or directory 151202 12:51:25 time=1449082285.467642 func=GetFmd level=WARN logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f171b4d3700 source=FmdSqlite:530 tident=<service> sec= uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found 151202 12:51:25 time=1449082285.467693 func=FSctl level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f171b4d3700 source=XrdFstOfs:1297 tident= sec=(null) uid=0 gid=0 name=- geo="" no fmd for fileid 136090623 on filesystem 65
#4 Updated by Gerard Bernabeu Altayo almost 5 years ago
- Status changed from New to Resolved
all this nodes have been retired now.