Project

General

Profile

Bug #10763

Retire old EOS nodes (cmseos2-5)

Added by Gerard Bernabeu Altayo almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Start date:
11/24/2015
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Duration:

Description

The following nodes should be retired: cmseos2,3,4,5. This are all 2x1Gbps nodes and very old.


I got the following email from David Fagan:

These are old
cmseos2,3,4,5 (disk 2009) 36TB

?
cmseos1 (2011) 108TB

-----
Getting there...

cmseos27 (2012) 162TB


For now I'd keep cmseos1 and 27, they both have 10GE NICs.

I think I'll need to write a small procedure on how to migrate disks.


Subtasks

Bug #10997: remove cmseos2-5 from the bestman gridftp list (maybe add some other nodes to it!)ResolvedGerard Bernabeu Altayo

History

#1 Updated by Gerard Bernabeu Altayo almost 4 years ago

Drain documentation available at http://eos.readthedocs.org/en/latest/configuration/draining.html

It does not make sense to make our own parallel documentation, I will just post in this ticket what I've done exactly for future reference:

[root@cmssrv222 ~]# eos fs ls -d

#....................................................................................................................................
#                   host (#...) #   id #           path #      drain #   progress #      files # bytes-left #  timeleft #retry #wopen
#....................................................................................................................................
[root@cmssrv222 ~]# eos fs ls | grep cmseos2
        cmseos2.fnal.gov (1095)     74   /storage/data1        default.0                        booted             rw      nodrain   online
        cmseos2.fnal.gov (1095)     75   /storage/data2        default.1                        booted             rw      nodrain   online
        cmseos2.fnal.gov (1095)     76   /storage/data3        default.3                        booted             rw      nodrain   online
       cmseos25.fnal.gov (1095)    103   /storage/data1        default.0                        booted             rw      nodrain   online
       cmseos25.fnal.gov (1095)    104   /storage/data2        default.1                        booted             rw      nodrain   online
       cmseos26.fnal.gov (1095)    105   /storage/data1        default.2                        booted             rw      nodrain   online
       cmseos26.fnal.gov (1095)    106   /storage/data2        default.3                        booted             rw      nodrain   online
       cmseos27.fnal.gov (1095)    107   /storage/data1        default.0                        booted             rw      nodrain   online
       cmseos27.fnal.gov (1095)    108   /storage/data2        default.1                        booted             rw      nodrain   online
       cmseos27.fnal.gov (1095)    109   /storage/data3        default.2                        booted             rw      nodrain   online
       cmseos28.fnal.gov (1095)    110   /storage/data1        default.0                        booted             rw      nodrain   online
       cmseos28.fnal.gov (1095)    111   /storage/data2        default.1                        booted             rw      nodrain   online
       cmseos29.fnal.gov (1095)    112   /storage/data1        default.2                        booted             rw      nodrain   online
       cmseos29.fnal.gov (1095)    113   /storage/data2        default.3                        booted             rw      nodrain   online
[root@cmssrv222 ~]# eos fs config 74 configstatus=drain
[root@cmssrv222 ~]# eos fs config 75 configstatus=drain
[root@cmssrv222 ~]# eos fs config 76 configstatus=drain
[root@cmssrv222 ~]# eos fs ls | grep cmseos3.fnal
        cmseos3.fnal.gov (1095)     71   /storage/data1        default.0                        booted             rw      nodrain   online
        cmseos3.fnal.gov (1095)     72   /storage/data2        default.2                        booted             rw      nodrain   online
        cmseos3.fnal.gov (1095)     73   /storage/data3        default.1                        booted             rw      nodrain   online
[root@cmssrv222 ~]# eos fs config 71 configstatus=drain
[root@cmssrv222 ~]# eos fs config 72 configstatus=drain
[root@cmssrv222 ~]# eos fs config 73 configstatus=drain
[root@cmssrv222 ~]# eos fs ls | grep cmseos4.fnal
        cmseos4.fnal.gov (1095)     68   /storage/data1        default.0                        booted             rw      nodrain   online
        cmseos4.fnal.gov (1095)     69   /storage/data2        default.1                        booted             rw      nodrain   online
        cmseos4.fnal.gov (1095)     70   /storage/data3        default.2                        booted             rw      nodrain   online
[root@cmssrv222 ~]# eos fs config 68 configstatus=drain
[root@cmssrv222 ~]# eos fs config 69 configstatus=drain
[root@cmssrv222 ~]# eos fs config 70 configstatus=drain
[root@cmssrv222 ~]# eos fs ls | grep cmseos5.fnal
        cmseos5.fnal.gov (1095)     65   /storage/data1        default.0                        booted             rw      nodrain   online
        cmseos5.fnal.gov (1095)     66   /storage/data2        default.1                        booted             rw      nodrain   online
        cmseos5.fnal.gov (1095)     67   /storage/data3        default.2                        booted             rw      nodrain   online
[root@cmssrv222 ~]# eos fs config 65 configstatus=drain
[root@cmssrv222 ~]# eos fs config 66 configstatus=drain
[root@cmssrv222 ~]# eos fs config 67 configstatus=drain
[root@cmssrv222 ~]# eos fs ls -d

#....................................................................................................................................
#                   host (#...) #   id #           path #      drain #   progress #      files # bytes-left #  timeleft #retry #wopen
#....................................................................................................................................
        cmseos5.fnal.gov (1095)     65   /storage/data1      prepare            0         0.00       0.00 B          50      0      4
        cmseos5.fnal.gov (1095)     66   /storage/data2      prepare            0         0.00       0.00 B          53      0      0
        cmseos5.fnal.gov (1095)     67   /storage/data3      prepare            0         0.00       0.00 B          57      0      4
        cmseos4.fnal.gov (1095)     68   /storage/data1      prepare            0         0.00       0.00 B          28      0      3
        cmseos4.fnal.gov (1095)     69   /storage/data2      prepare            0         0.00       0.00 B          32      0      1
        cmseos4.fnal.gov (1095)     70   /storage/data3      prepare            0         0.00       0.00 B          36      0      2
        cmseos3.fnal.gov (1095)     71   /storage/data1      prepare            0         0.00       0.00 B           3      0      5
        cmseos3.fnal.gov (1095)     72   /storage/data2      prepare            0         0.00       0.00 B           7      0      0
        cmseos3.fnal.gov (1095)     73   /storage/data3      prepare            0         0.00       0.00 B          10      0      2
        cmseos2.fnal.gov (1095)     74   /storage/data1     draining            0     143.51 k      7.59 TB       86298      0      0
        cmseos2.fnal.gov (1095)     75   /storage/data2     draining            0     138.33 k      7.53 TB       86324      0      2
        cmseos2.fnal.gov (1095)     76   /storage/data3     draining            0     152.90 k      8.09 TB       86328      0      2
[root@cmssrv222 ~]# 

#2 Updated by Gerard Bernabeu Altayo almost 4 years ago

  • Subject changed from Retire old EOS nodes to Retire old EOS nodes (cmseos2-5)

The drain expired for some nodes, resubmitted it with:

[root@cmssrv222 ~]# for i in `seq 65 76`; do eos fs config $i configstatus=drain; done

#3 Updated by Gerard Bernabeu Altayo almost 4 years ago

All nodes have some issues with files, focusing on cmseos5 first I think that there is inevitable data loss on it:

EOS Console [root://localhost] |/eos/> fs ls -d

#....................................................................................................................................
#                   host (#...) #   id #           path #      drain #   progress #      files # bytes-left #  timeleft #retry #wopen
#....................................................................................................................................
        cmseos5.fnal.gov (1095)     65   /storage/data1     stalling            0       190.00    284.26 MB 99999999999      0      0
        cmseos5.fnal.gov (1095)     66   /storage/data2     stalling            0       251.00      3.50 GB 99999999999      0      0
        cmseos5.fnal.gov (1095)     67   /storage/data3     stalling            0       240.00      2.84 GB 99999999999      0      0
EOS Console [root://localhost] |/eos/> fs status 65
# ------------------------------------------------------------------------------------
# FileSystem Variables
# ....................................................................................
bootcheck                        := 0
configstatus                     := drain
headroom                         := 5000000000
host                             := cmseos5.fnal.gov
hostport                         := cmseos5.fnal.gov:1095
id                               := 65
path                             := /storage/data1
port                             := 1095
queue                            := /eos/cmseos5.fnal.gov:1095/fst
queuepath                        := /eos/cmseos5.fnal.gov:1095/fst/storage/data1
scaninterval                     := 2419200
schedgroup                       := default.0
stat.active                      := online
stat.balancer.running            := 0
stat.boot                        := booted
stat.disk.bw                     := 144.000000
stat.disk.iops                   := 69
stat.disk.load                   := 0.000000
stat.disk.readratemb             := 0.000000
stat.disk.writeratemb            := 0.001160
stat.drain                       := stalling
stat.drainbytesleft              := 284262400
stat.drainer                     := on
stat.drainer.running             := 0
stat.drainfiles                  := 190
stat.drainprogress               := 0
stat.drainretry                  := 0
stat.fsck.d_cx_diff              := 1
stat.fsck.d_mem_sz_diff          := 1
stat.fsck.d_sync_n               := 8071
stat.fsck.m_cx_diff              := 0
stat.fsck.m_mem_sz_diff          := 0
stat.fsck.m_sync_n               := 8011
stat.fsck.mem_n                  := 8071
stat.fsck.orphans_n              := 0
stat.fsck.rep_diff_n             := 1
stat.fsck.rep_missing_n          := 0
stat.fsck.unreg_n                := 1
stat.geotag                      := 
stat.heartbeattime               := 1449082130
stat.net.ethratemib              := 238.000000
stat.net.inratemib               := 0.001370
stat.net.outratemib              := 0.001550
stat.nominal.filled              := 0.000000
stat.ropen                       := 0
stat.ropen.hotfiles              :=  
stat.statfs.bavail               := 2929988712
stat.statfs.bfree                := 2929988712
stat.statfs.blocks               := 2930058112
stat.statfs.bsize                := 4096
stat.statfs.capacity             := 12001518026752
stat.statfs.ffree                := 11720342300
stat.statfs.files                := 11720363520
stat.statfs.filled               := 0.002369
stat.statfs.freebytes            := 12001233764352
stat.statfs.fused                := 86917120
stat.statfs.namelen              := 255
stat.statfs.type                 := 1481003842
stat.statfs.usedbytes            := 284262400
stat.timeleft                    := 99999999999
stat.usedfiles                   := 8071
stat.wopen                       := 0
stat.wopen.hotfiles              :=  
uuid                             := e5fb4e25-0ca8-49dc-b5c0-578ea761637b
# ....................................................................................
# Risk Analysis
# ....................................................................................
number of files                  :=        190 (100.00%)
files healthy                    :=          0 (0.00%)
files at risk                    :=        102 (53.68%)
files inaccessbile               :=         88 (46.32%)
files pending deletion           :=          0
# ------------------------------------------------------------------------------------
EOS Console [root://localhost] |/eos/> fs dumpmd 65 -path
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_20/cache/nll_der-toys:0.0-wp2000-08800fe638.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_5/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_27/nll_der-toys:0.033343-wp2500-fc6f053826.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_21/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_18/plots/limit_band_plot-log-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_7/nll_der-toys:0.0-wp2000-2cf1fa674e.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_17/cache/nll_der-toys:0.0-wp3000-709e191ee4.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_18/cache/nll_der-toys:0.0-wp3000-6ad6ecf85a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_19/cache/nll_der-toys:0.0-wp2500-492e65cad2.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_17/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_22/nll_der-toys:0.0-wp2500-3aeb8251df.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_23/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_4/cache/nll_der-toys:0.0-wp2500-77680253b0.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_16/nll_der-toys:0.0-wp2500-9b3e4fe432.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_12/cache/quant-toys:0-wp2000-55ee068e8d.db
path=/lost+found/orphans/2261676/c7592d32-59ad-11e5-bbc8-001e67abf518-158-0-logArchive.tar.gz.125305362
path=/lost+found/orphans/2260531/31eddb48-59b3-11e5-bbc8-001e67abf518-60-0-logArchive.tar.gz.125175327
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_2/cache/nll_der-toys:0.0-wp2500-892cfe55de.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_4.py
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_12/model_summary_general.thtml
path=/lost+found/orphans/2261093/4e61074a-59dc-11e5-adda-a0369f23d01e-942-0-logArchive.tar.gz.125222436
path=/lost+found/orphans/2261093/4e61074a-59dc-11e5-adda-a0369f23d01e-957-0-logArchive.tar.gz.125231140
path=/lost+found/orphans/2261117/4e61074a-59dc-11e5-adda-a0369f23d01e-132-0-logArchive.tar.gz.125247526
path=/lost+found/orphans/2266398/752830fe-5b87-11e5-9b56-02163e00d7b3-160-0-logArchive.tar.gz.125677607
path=/lost+found/orphans/2260548/31eddb48-59b3-11e5-bbc8-001e67abf518-869-0-logArchive.tar.gz.125166122
path=/lost+found/orphans/2260219/55d5b21c-59d9-11e5-a196-a0369f23d050-27-0-logArchive.tar.gz.125176877
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_2/quant-toys:0-wp2000-e0de1a86cc.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_2/cache/nll_der-toys:0.114590-wp2000-cde86f9252.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5/cache/nll_der-toys:0.0-wp2000-33e19f5499.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_18/model_summary_rates.thtml
path=/lost+found/orphans/2260979/81162ad6-59fe-11e5-a196-a0369f23d050-9-0-logArchive.tar.gz.125210674
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1743-0-logArchive.tar.gz.125236285
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_2/cache/nll_der-toys:0.114590-wp2000-ca9b54a228.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_23/quant-toys:0-wp3000-c0fdb49949.cfg
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1310-0-logArchive.tar.gz.125223488
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_8/quant-toys:0-wp2000-1603ad9e10.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_19/cache/nll_der-toys:0.0-wp3000-52f265da48.cfg
path=/lost+found/orphans/2266398/1fd19ff6-5b86-11e5-9b56-02163e00d7b3-182-0-logArchive.tar.gz.125659209
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_11/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_25/cache/nll_der-toys:0.010756-wp3000-654bfc0c28.cfg
path=/lost+found/orphans/2260979/d37b909c-5a01-11e5-a196-a0369f23d050-9-0-logArchive.tar.gz.125213267
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_7/quant-toys:0-wp2000-078ebde551.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_8/nll_der-toys:0.0-wp3000-3d4bd1084b.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_26/cache/nll_der-toys:0.010756-wp3000-e24971f6e6.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_18/nll_der-toys:0.0-wp2000-2c1044e6e9.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_22/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/muon_ge1btags_IDMLJet350Lep150MET100_discovery.json
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_24/cache/nll_der-toys:0.114590-wp2000-09bb227562.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_23/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_1/cache/nll_der-toys:0.0-wp2000-c37adecc24.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_1/cache/nll_der-toys:0.010756-wp3000-7f46bd0c74.db
path=/lost+found/orphans/2260658/6c80894a-59c2-11e5-bbc8-001e67abf518-703-0-logArchive.tar.gz.125187177
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_14/nll_der-toys:0.010756-wp3000-ede352e3bb.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_23/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_12.stdout
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_19/nll_der-toys:0.0-wp2000-c47123d2ad.cfg
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015C-PromptReco-v1.DoubleMuon/jobExecCondor_Run2015C-PromptReco-v1.DoubleMuon_160_RA2AnalysisTree_exp_302723.stdout
path=/eos/uscms/store/user/ptan/work/sl6/development/CMSSW_7_1_20/src/run_0002/work_10000_3934/powhegbox_Z/pwhg_checklimits
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_12/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_7/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150807_204321/0000/B2GEDMNtuple_182.root
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_19/quant-toys:0-wp3000-d7c940dfa2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_17/cache/quant-toys:0-wp3000-5c948e1f9b.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_11/nll_der-toys:0.0-wp3000-2854f18738.cfg
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1241-0-logArchive.tar.gz.125227666
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_13/cache/nll_der-toys:0.0-wp3000-da74ec1b17.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_18/cache/quant-toys:0-wp2500-3e2c957b6a.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_16/quant-toys:0-wp2000-21e186f39d.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_3/nll_der-toys:0.0-wp2500-9bc3e22fe3.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_1/cache/nll_der-toys:0.0-wp2500-dadea2ba5c.db
path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-571-0-logArchive.tar.gz.125182625
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_2/cache/nll_der-toys:0.033343-wp2500-f8444f4c5b.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_19.stdout
path=/lost+found/orphans/2179013/cmsRun_14.log.tar.gz.120396459
path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150807_204321/0000/log/cmsRun_105.log.tar.gz
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1666-0-logArchive.tar.gz.125234346
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_20/cache/nll_der-toys:0.114590-wp2000-d3ff2f68d2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_20/nll_der-toys:0.0-wp2000-742b9015af.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_8/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_11/cache/nll_der-toys:0.0-wp2500-042cc09f4a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_27/cache/nll_der-toys:0.0-wp2000-6b8ec22d15.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_10/nll_der-toys:0.033343-wp2500-ad3ee63d5f.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/quant-toys:0-wp2500-960fc6ef76.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_9/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_12/cache/nll_der-toys:0.0-wp2000-b8d96aa8e2.db
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015C-PromptReco-v1.JetHT/jobExecCondor_Run2015C-PromptReco-v1.JetHT_212_RA2AnalysisTree_exp_303823.stderr
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_143_RA2AnalysisTree_exp_230053.condor
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_24/nll_der-toys:0.033343-wp2500-a38cf86a78.cfg
path=/lost+found/orphans/2260469/89620134-59ad-11e5-bbc8-001e67abf518-1110-0-logArchive.tar.gz.125145807
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/electron_ge1btags_IDTLJet250Lep150MET50_discovery.json
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_26/model_summary_rates.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_24/cache/quant-toys:0-wp2500-9e44a880a5.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_24/nll_der-toys:0.0-wp2000-413b0f39e1.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/quant-toys:0-wp2500-83255dd362.db
path=/lost+found/orphans/2260469/ce14512a-59b1-11e5-bbc8-001e67abf518-248-0-logArchive.tar.gz.125160152
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_25/cache/nll_der-toys:0.033343-wp2500-b968cc1080.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_27/cache/quant-toys:0-wp2500-0890c5e533.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_23/cache/nll_der-toys:0.0-wp2000-7f03519e3a.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_2/nll_der-toys:0.0-wp2500-35b193a804.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_15/quant-toys:0-wp2000-4b5e3c2807.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_22/cache/nll_der-toys:0.0-wp2000-50aedc77a1.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_18/cache/nll_der-toys:0.0-wp3000-ded9d44ab2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_20/nll_der-toys:0.033343-wp2500-7a8fe444c2.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/electron_bayesian_limits_expected_ge1btags_IDTTJet250Lep120MET50.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_21/nll_der-toys:0.114590-wp2000-397c5ab425.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/quant-toys:0-wp2500-83255dd362.cfg
path=/lost+found/orphans/2260979/098edc8c-5a04-11e5-a196-a0369f23d050-20-0-logArchive.tar.gz.125214997
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_15/cache/nll_der-toys:0.0-wp2500-830ddf6a3a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_10/cache/quant-toys:0-wp2500-e8b4ae4b04.db
path=/lost+found/orphans/2260979/16da51ee-5a01-11e5-a196-a0369f23d050-24-0-logArchive.tar.gz.125212448
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_2/nll_der-toys:0.010756-wp3000-b9ebd89fdc.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_3/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_25/cache/quant-toys:0-wp3000-9ec7dd6f7d.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_13/cache/nll_der-toys:0.0-wp2000-1eea988edb.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_14/cache/nll_der-toys:0.0-wp3000-dd7af868cb.cfg
path=/lost+found/orphans/2260837/c5571204-59dc-11e5-a196-a0369f23d050-31-0-logArchive.tar.gz.125183286
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_12.condor
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_19/cache/quant-toys:0-wp2500-94ec3f53d8.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_6/quant-toys:0-wp3000-2cdf240713.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_2/plots/elec_invmass_ge1btags_stack.png
path=/lost+found/orphans/2404866/Spring15.TTJets_HT-800to1200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8_Expectation.root.134833998
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.MET/jobExecCondor_Run2015D-PromptReco-v3.MET_23_RA2AnalysisTree_exp_276208.stderr
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_10/plots/limit_band_plot-bayesian.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_2/plots/elec_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_45_RA2AnalysisTree_exp_230093.condor
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_18/cache/nll_der-toys:0.033343-wp2500-af448fdb44.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_6/model_summary_rates.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_8.condor
path=/lost+found/orphans/2261079/4e61074a-59dc-11e5-adda-a0369f23d01e-1618-0-logArchive.tar.gz.125225831
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_14/nll_der-toys:0.0-wp2500-3951cd37ee.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_16/cache/quant-toys:0-wp2500-252ee6b98b.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_6/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_3/nll_der-toys:0.010756-wp3000-ea4c2607fd.cfg
path=/lost+found/orphans/2260598/fa0f2974-59bd-11e5-bbc8-001e67abf518-223-0-logArchive.tar.gz.125187442
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_18/index.html
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_5/cache/nll_der-toys:0.114590-wp2000-9663860664.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_7/model_summary_nuisance.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_20/quant-toys:0-wp3000-59ee280091.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/muon_bayesian_limits_expected_ge1btags_IDMTJet350Lep120MET100.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_25/nll_der-toys:0.0-wp2500-b7bb2c1c5f.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_14/cache/nll_der-toys:0.010756-wp3000-da6ecf89d3.cfg
path=/eos/uscms/store/user/pilot/JetHT/B2GAnaFW_V74x_V43/150805_210241/0000/failed/log/cmsRun_35.log.tar.gz
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_20/cache/quant-toys:0-wp3000-7d758a0ac8.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_23/nll_der-toys:0.0-wp2500-9541ba1e45.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_19/cache/nll_der-toys:0.114590-wp2000-af28f86990.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_21/cache/quant-toys:0-wp3000-9d95c2212a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/electron_ge1btags_IDTTJet250Lep120MET100_discovery.json
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_25/cache/nll_der-toys:0.033343-wp2500-b78726f0e7.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_18/nll_der-toys:0.033343-wp2500-b8addcf58a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/electron_bayesian_limits_expected_ge1btags_IDTLJet450Lep120MET50.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_4/cache/nll_der-toys:0.0-wp3000-cd4c3112fd.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_14/plots/mu_invmass_ge1btags_stack.png
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_23/nll_der-toys:0.114590-wp2000-129573224a.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_17/cache/nll_der-toys:0.0-wp3000-2838ccb834.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_25/quant-toys:0-wp3000-34fb42cbdc.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_18/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_9/model_plots.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_ML_3fb/Muon_ML_3fb_3/cache/nll_der-toys:0.033343-wp2500-c487654e3e.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3/nll_der-toys:0.010756-wp3000-b856611bdb.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_3.condor
path=/lost+found/orphans/2180880/cmsRun_47.log.tar.gz.120497592
path=/lost+found/orphans/2260548/186434d6-59ba-11e5-bbc8-001e67abf518-108-0-logArchive.tar.gz.125177266
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_4/cache/nll_der-toys:0.114590-wp2000-1ec6a96890.cfg
path=/lost+found/orphans/2179013/cmsRun_46.log.tar.gz.120394179
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/electron_bayesian_limits_expected_ge1btags_IDMLJet450Lep150MET150.txt
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_24/cache/quant-toys:0-wp3000-aa781ee72c.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_5.condor
path=/eos/uscms/store/user/lpcmbja/noreplica/zzDR_p0plusp1_050915_correlatedBkg_sigThrsld2.5_deCorrdMedHighNuis_trendGuessObserved_snapshot_fix/condor.VVVh_CMS_common_expected_minuit2_0p0005_tries5_strategy1.log
path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-522-0-logArchive.tar.gz.125193169
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_6/cache/nll_der-toys:0.0-wp2000-0ddfeb4e7d.cfg
path=/lost+found/orphans/2260219/3f9c812a-59d3-11e5-a196-a0369f23d050-29-0-logArchive.tar.gz.125171158
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_ML_3fb/Elec_ML_3fb_24/cache/nll_der-toys:0.114590-wp2000-09bb227562.db
path=/lost+found/orphans/2260749/a50a2922-59c9-11e5-bbc8-001e67abf518-286-0-logArchive.tar.gz.125191132
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_14/cache/quant-toys:0-wp2000-fef6a41b4c.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_27.csh
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TT_3fb/Muon_TT_3fb_17/cache/quant-toys:0-wp2500-78635dd5c4.cfg
path=/eos/uscms/store/user/jbradmil/lostlep/logs/Run2015D-PromptReco-v3.JetHT/jobExecCondor_Run2015D-PromptReco-v3.JetHT_84_RA2AnalysisTree_exp_230132.stdout
path=/lost+found/orphans/2260469/89620134-59ad-11e5-bbc8-001e67abf518-1241-0-logArchive.tar.gz.125152227
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_TL_3fb/Muon_TL_3fb_8/cache/nll_der-toys:0.0-wp2000-68eb4a697e.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_24/nll_der-toys:0.0-wp2500-185549eb2f.cfg
path=/lost+found/orphans/2260979/a5e483f0-59fc-11e5-a196-a0369f23d050-13-0-logArchive.tar.gz.125209064
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_15/cache/nll_der-toys:0.0-wp2000-7af3e0c80e.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Muon_MT_3fb/Muon_MT_3fb_15/model_summary_rate_impacts.thtml
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_17/model_summary_general.thtml
path=/lost+found/orphans/2260548/31eddb48-59b3-11e5-bbc8-001e67abf518-730-0-logArchive.tar.gz.125180918
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TL_3fb/Elec_TL_3fb_3/cache/nll_der-toys:0.114590-wp2000-d68574c190.cfg
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_TT_3fb/Elec_TT_3fb_14/cache/nll_der-toys:0.0-wp2000-c3e1f98183.db
path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
EOS Console [root://localhost] |/eos/> file info /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
  File: '/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg'  Flags: 0644
  Size: 7205
Modify: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.100891000
Change: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.89492856
  CUid: 47452 CGid: 5063  Fxid: 081c93ff Fid: 136090623    Pid: 2434745   Pxid: 002526b9
XStype: adler    XS: d1 a1 8a 64     ETAG: 36531548442329088:d1a18a64
replica Stripes: 2 Blocksize: 4k LayoutId: 00100112
  #Rep: 2
 #   fs-id  #...................................................................................................................................
            #                   host  #     schedgroup #           path #     boot # configstatus #      drain # active #                 geotag
            #...................................................................................................................................
  0     107        cmseos27.fnal.gov         default.0   /storage/data1     booted             rw      nodrain   online                         
  1      65         cmseos5.fnal.gov         default.0   /storage/data1     booted          drain     stalling   online                         
*******
EOS Console [root://localhost] |/eos/> file check /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
path="/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg" fid="081c93ff" size="7205" nrep="2" checksumtype="adler" checksum="d1a18a6400000000000000000000000000000000" 
error: unable to retrieve file meta data from cmseos27.fnal.gov:1095 [61]
error: unable to retrieve file meta data from cmseos5.fnal.gov:1095 [61]
EOS Console [root://localhost] |/eos/> file verify /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
success: sending verify to fsid= 107 for path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
success: sending verify to fsid= 65 for path=/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
EOS Console [root://localhost] |/eos/> file check /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
path="/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg" fid="081c93ff" size="7205" nrep="2" checksumtype="adler" checksum="d1a18a6400000000000000000000000000000000" 
error: unable to retrieve file meta data from cmseos27.fnal.gov:1095 [61]
error: unable to retrieve file meta data from cmseos5.fnal.gov:1095 [61]
EOS Console [root://localhost] |/eos/> file info /eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg
  File: '/eos/uscms/store/user/drankin/Theta_Oct13/Elec_MT_3fb/Elec_MT_3fb_9/cache/nll_der-toys:0.114590-wp2000-5723c02c49.cfg'  Flags: 0644
  Size: 7205
Modify: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.100891000
Change: Wed Oct 14 05:54:23 2015 Timestamp: 1444820063.89492856
  CUid: 47452 CGid: 5063  Fxid: 081c93ff Fid: 136090623    Pid: 2434745   Pxid: 002526b9
XStype: adler    XS: d1 a1 8a 64     ETAG: 36531548442329088:d1a18a64
replica Stripes: 2 Blocksize: 4k LayoutId: 00100112
  #Rep: 2
 #   fs-id  #...................................................................................................................................
            #                   host  #     schedgroup #           path #     boot # configstatus #      drain # active #                 geotag
            #...................................................................................................................................
  0     107        cmseos27.fnal.gov         default.0   /storage/data1     booted             rw      nodrain   online                         
  1      65         cmseos5.fnal.gov         default.0   /storage/data1     booted          drain     stalling   online                         
*******
EOS Console [root://localhost] |/eos/> 

This basically says this file does NOT exist on the FSTs, on the FST I see consistent errors:


151202 12:50:44 8029 XrootdXeq: root.27203:82@cmssrv222 login as daemon
151202 12:50:44 8029 FstOfs_stat: root.27203:82@cmssrv222 Unable to stat file /storage/data1/00003529/081c93ff; no such file or directory
151202 12:50:44 time=1449082244.465897 func=GetFmd                   level=WARN  logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17bebc3700 source=FmdSqlite:530                  tident=<service> sec=      uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found
151202 12:50:44 time=1449082244.465941 func=FSctl                    level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17bebc3700 source=XrdFstOfs:1297                 tident= sec=(null) uid=0 gid=0 name=- geo="" no fmd for fileid 136090623 on filesystem 65
151202 12:51:07 time=1449082267.380189 func=GetFmd                   level=WARN  logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=FmdSqlite:530                  tident=<service> sec=      uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found
151202 12:51:07 time=1449082267.380255 func=Verify                   level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=Verify:108                     tident= sec=(null) uid=0 gid=0 name=- geo="" unable to verify file id=81c93ff on fs=65 path=/storage/data1/00003529/081c93ff - stat on local disk failed
151202 12:51:07 time=1449082267.380282 func=GetFmd                   level=WARN  logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=FmdSqlite:530                  tident=<service> sec=      uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found
151202 12:51:07 time=1449082267.380305 func=Verify                   level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f17b66fa700 source=Verify:122                     tident= sec=(null) uid=0 gid=0 name=- geo="" unable to verify id=81c93ff on fs=65 path=/storage/data1/00003529/081c93ff - no local MD stored
151202 12:51:25 11252 FstOfs_stat: root.27203:82@cmssrv222 Unable to stat file /storage/data1/00003529/081c93ff; no such file or directory
151202 12:51:25 time=1449082285.467642 func=GetFmd                   level=WARN  logid=CommonFmdSqliteHandler unit=fst@cmseos5.fnal.gov:1095 tid=00007f171b4d3700 source=FmdSqlite:530                  tident=<service> sec=      uid=0 gid=0 name= geo="" unable to get fmd for fid 136090623 on fs 65 - record not found
151202 12:51:25 time=1449082285.467693 func=FSctl                    level=ERROR logid=static.............................. unit=fst@cmseos5.fnal.gov:1095 tid=00007f171b4d3700 source=XrdFstOfs:1297                 tident= sec=(null) uid=0 gid=0 name=- geo="" no fmd for fileid 136090623 on filesystem 65

#4 Updated by Gerard Bernabeu Altayo over 3 years ago

  • Status changed from New to Resolved

all this nodes have been retired now.



Also available in: Atom PDF