Project

General

Profile

Task #11004

Handling failed encp transfer due to configuration change

Added by Chih-Hao Huang about 4 years ago. Updated about 4 years ago.

Status:
Resolved
Priority:
Normal
Start date:
11/24/2015
Due date:
11/26/2015
% Done:

100%

Estimated time:
2.00 h
Spent time:
Duration: 3

Description

After configuration change by DMS, we saw bunch of "EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set:" errors.

This is a known bug that, in our version 2.2, when encp fails and does the clean up, layer one can not be removed from /pnfs interface.

These need to be identified and be manually cleaned up in chimera database.

History

#1 Updated by Chih-Hao Huang about 4 years ago

  • % Done changed from 0 to 20

psql (8.4.4, server 9.3.9)
WARNING: psql version 8.4, server version 9.3.
Some psql features might not work.
Type "help" for help.

accounting=> select distinct error from encp_error where date > '2015-11-24 16:00:00' and node like 'cmsstor%';
error

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15GenOnly/DYToLL_1J_8TeV-amcatnloFXFX-pythia8/GEN/MCRUN2_71_V1-v1/20000/744108C2-F67E-E511-9E42-842B2B2AB616.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15FSPremix/SMS-T1tttt_mGluino-625to650_mLSP-200to400_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/AODSIM/MCRUN2_74_V9-v1/10000/4E010DA2-C07C-E511-BEFB-02163E0F3C7.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/WJetsToLNu_HT-800To1200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/LHE/MCRUN2_71_V1_ext1-v1/50000/24076045-6F6F-E511-8215-008CFA05E874.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/DYToLL_2J_8TeV-amcatnloFXFX-pythia8/LHE/MCRUN2_71_V1-v1/80000/CA8AB7E2-D77E-E511-9891-44A8424A1D64.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/DYToEE_NNPDF30_13TeV-powheg-pythia8/MINIAODSIM/74X_mcRun2_asymptotic_v2-v1/50000/F25FB014-746F-E511-B7E7-39F343D4313A.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISummer15GS/QCD_Pt-20toInf_MuEnrichedPt15_TuneCUETP8M1_13TeV_pythia8/GEN-SIM/MCRUN2_71_V1-v1/20001/3473E89D-7D7C-E511-A6E4-FA163EA84A3A.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/Summer12DR53X/TToLeptons_top_t-channel_TuneZ2Star_8TeV_4f-powheg-tauola/AODSIM/PU_S10_START53_V19-v1/20000/3274DFA5-847C-E511-84A5-90B11C08CDB7.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/DYJetsToNuNu_TuneCUETP8M1_13TeV-amcatnloFXFX-pythia8/MINIAODSIM/74X_mcRun2_asymptotic_v2-v1/30000/505AC0DB-E16D-E511-9CFA-003048FFCC1E.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISummer15GS/MinBias_TuneCUETP8M1_13TeV-pythia8/GEN-SIM/MCRUN2_71_V1-v2/80000/5CEECBAB-2577-E511-984F-0025904C67B4.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15GenOnly/DYToLL_0J_8TeV-amcatnloFXFX-pythia8/GEN/MCRUN2_71_V1-v1/10000/A8DECF46-E97D-E511-8175-0CC47A13D416.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/DYToLL_1J_8TeV-amcatnloFXFX-pythia8/LHE/MCRUN2_71_V1-v1/10000/FACC1466-C17E-E511-8DE2-D4AE52806A65.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/TT_TuneCUETP8M1_13TeV-amcatnlo-pythia8/MINIAODSIM/74X_mcRun2_asymptotic_v2-v1/50000/E64A8205-4A70-E511-BF24-0002C90B7FC8.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/TTJets_HT-800to1200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/LHE/MCRUN2_71_V1_ext1-v1/10000/DCFA9A37-786F-E511-A70F-0CC47A0093B6.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/JpsiToMuMu_OniaMuonFilter_TuneCUEP8M1_13TeV-pythia8/MINIAODSIM/Asympt50ns_74X_mcRun2_asymptotic50ns_v0-v1/60000/EE3F6BD5-4E74-E511-91A5-E41DD08DE30.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/ST_t-channel_4f_mtop1695_leptonDecays_13TeV-amcatnlo-pythia8_TuneCUETP8M1/MINIAODSIM/74X_mcRun2_asymptotic_v2-v1/30000/4CC01DA7-BF6E-E511-86
1-A01D48EE8212.root
EEXIST: [ ERRNO 17 ] File exists: None
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/DYToLL_1J_8TeV-amcatnloFXFX-pythia8/LHE/MCRUN2_71_V1-v1/20000/E2A140AB-E37E-E511-B93F-00A0D1EE8D7C.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15DR74/TT_TuneCUETP8M1_13TeV-powheg-pythia8/AODSIM/Startup25ns_EXOReReco_74X_Spring15_mcRun2_startup25ns_v0-v1/50000/60AC2E3A-9D79-E511-869A-001E6750478.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/DYToLL_2J_8TeV-amcatnloFXFX-pythia8/LHE/MCRUN2_71_V1-v1/80001/F2254C30-E27E-E511-81CD-90B11C267182.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/WWTo4Q_4f_13TeV_amcatnloFXFX_madspin_pythia8/MINIAODSIM/74X_mcRun2_asymptotic_v2-v1/10000/A83D5156-E06E-E511-854B-001E67A3FE66.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15GenOnly/DYToLL_1J_8TeV-amcatnloFXFX-pythia8/GEN/MCRUN2_71_V1-v1/20000/8CD0D5ED-FB7E-E511-9D82-00259073E49A.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/DYToLL_1J_8TeV-amcatnloFXFX-pythia8/LHE/MCRUN2_71_V1-v1/20006/1220E5A6-FE7E-E511-8BAA-008CFA1111EC.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISummer15GS/TT_TuneCUETP8M1_13TeV-powheg-scaledown-pythia8/GEN-SIM/MCRUN2_71_V1-v1/20000/5C683984-2F85-E511-8060-0CC47A6C1866.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15GenOnly/DYToLL_2J_8TeV-amcatnloFXFX-pythia8/GEN/MCRUN2_71_V1-v1/80000/FA459F65-F57E-E511-A0E6-001E67399C89.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/ZZTo4Q_13TeV_amcatnloFXFX_madspin_pythia8/MINIAODSIM/74X_mcRun2_asymptotic_v2-v1/60000/A03F6F38-FF6E-E511-ABE3-002354EF3BD0.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/DYToLL_1J_8TeV-amcatnloFXFX-pythia8/LHE/MCRUN2_71_V1-v1/10001/5C1AF3D5-CD7E-E511-A129-00259090831A.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIIWinter15wmLHE/DYToLL_1J_8TeV-amcatnloFXFX-pythia8/LHE/MCRUN2_71_V1-v1/20008/840630E3-237F-E511-B3FB-0025904C7F5E.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/data/Run2015E/MinimumBias18/RAW/v1/000/262/167/00000/00BA980E-E68F-E511-A9EF-02163E014118.root
EEXIST: [ ERRNO 17 ] File exists: Layer 1 is already set: /pnfs/fs/usr/cms/WAX/11/store/mc/RunIISpring15MiniAODv2/QCD_HT100to200_TuneCUETP8M1_13TeV-madgraphMLM-pythia8/MINIAODSIM/74X_mcRun2_asymptotic_v2-v1/60001/DCE21675-DE6C-E511-B7F4-0025905A48D0.root
(29 rows)

accounting=>

#2 Updated by Chih-Hao Huang about 4 years ago

[root@cmsphedex-disk HUANGCH]# for i in `cat t-2015-11-24`

do
enstore pnfs --id $i
done | tee pnfsid-2015-11-24

00002FD50ADDA5D94D0780460FB41FEA8E07
0000EA8FC0046F40498ABCF56C40357F4D95
00005C4FF2B3F1724FC9A2A03DEE1BE47869
000063E5C3A5392948B78838F597BA902DCA
0000798E634D5256464BBBFC3B57852A7F01
00000903E09A16D84B6593C81F8A56182AA7
000088095A108386418FB3771A93E4C93357
0000378765762BDD4924AB08BCA1245A8357
00008A6C8CA0A4D144F09F2A8D401F89A40C
0000432CAB2848174B9CB45771B1E4DBA544
000015F22319051D4E7C9D58590C68FA5446
0000FC055F51ECF74F27BA4119147221BE0A
000059B5E989EA9E4C019727BA79954B5F4A
000005FDACDF0081442DA9E28B713AB515F7
000084BEA317D3544F0C953841EFEB258D44
000067F9EF42C13C4E18AAC34A5948A2C857
0000FBD39840DFF34DFEA332F202D8E07EAC
0000A1D22F1C82A340CBA3161CD6981FD4FC
0000BE834BEF72444ABDA85CA4633F51634E
0000E97D3B635C894B31A5F2F69498B98B8C
0000BFD034BB9CE04C1FB6DFCF2CCC3A26F2
0000509E43E1B60143E28935BE860B91F664
000019E745545F3A4F56A81612303F93FDDA
0000EB0C0866953E4453AEC797504BF6358D
000019834567F5884982BECFDF2EE23D1EC4
00002D59E161AF7243148F13A04AEA13E730
000082B34996C86E4480A5CC5ACAFFF95940
0000D483C26E08E341468510E989037E5A00

#3 Updated by Chih-Hao Huang about 4 years ago

  • % Done changed from 20 to 60

[root@cmsphedex-disk HUANGCH]# for i in `cat t-2015-11-24`; do enstore pnfs --cat $i 1; done | tee bfid-2015-11-24
CDMS144808126800000
CDMS144787574300007
CDMS144789020700001
CDMS144814022100000
CDMS144797131000000
CDMS144806217000001
CDMS144801865200000
CDMS144797505700006
CDMS144784526400008
CDMS144809144600004
CDMS144808034700000
CDMS144792995600003
CDMS144786185000001
CDMS144792666500000
CDMS144789887700008
CDMS144808160900003
CDMS144810537200001
CDMS144814346900000
CDMS144797556000002
CDMS144809048300001
CDMS144808379400007
CDMS144803303700000
CDMS144814290500001
CDMS144797522000008
CDMS144808897800005
CDMS144807968100000
CDMS144825111100000
CDMS144800464300000

sh-3.2$ for i in `cat bfid-2015-11-24`

do
enstore info --bfid $i | grep deleted
done

'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
'deleted': 'yes',
sh-3.2$ exit

#4 Updated by Chih-Hao Huang about 4 years ago

  • Status changed from New to Assigned
  • % Done changed from 60 to 90

[root@cmschimera HUANGCH]# for i in `cat pnfsid-2015-11-24`; do echo "delete from t_level_1 where ipnfsid='$i';"; echo "delete from t_level_4 where ipnfsid='$i';"; echo "delete from t_storageinfo where ipnfsid='$i';"; echo "delete from t_locationinfo where ipnfsid='$i' and itype=0;"; done > cmd1
[root@cmschimera HUANGCH]# more cmd1
delete from t_level_1 where ipnfsid='00002FD50ADDA5D94D0780460FB41FEA8E07';
delete from t_level_4 where ipnfsid='00002FD50ADDA5D94D0780460FB41FEA8E07';
delete from t_storageinfo where ipnfsid='00002FD50ADDA5D94D0780460FB41FEA8E07';
delete from t_locationinfo where ipnfsid='00002FD50ADDA5D94D0780460FB41FEA8E07' and itype=0;
delete from t_level_1 where ipnfsid='0000EA8FC0046F40498ABCF56C40357F4D95';
delete from t_level_4 where ipnfsid='0000EA8FC0046F40498ABCF56C40357F4D95';
delete from t_storageinfo where ipnfsid='0000EA8FC0046F40498ABCF56C40357F4D95';
delete from t_locationinfo where ipnfsid='0000EA8FC0046F40498ABCF56C40357F4D95' and itype=0;
delete from t_level_1 where ipnfsid='00005C4FF2B3F1724FC9A2A03DEE1BE47869';
delete from t_level_4 where ipnfsid='00005C4FF2B3F1724FC9A2A03DEE1BE47869';
delete from t_storageinfo where ipnfsid='00005C4FF2B3F1724FC9A2A03DEE1BE47869';
delete from t_locationinfo where ipnfsid='00005C4FF2B3F1724FC9A2A03DEE1BE47869' and itype=0;
delete from t_level_1 where ipnfsid='000063E5C3A5392948B78838F597BA902DCA';
delete from t_level_4 where ipnfsid='000063E5C3A5392948B78838F597BA902DCA';
delete from t_storageinfo where ipnfsid='000063E5C3A5392948B78838F597BA902DCA';
delete from t_locationinfo where ipnfsid='000063E5C3A5392948B78838F597BA902DCA' and itype=0;
delete from t_level_1 where ipnfsid='0000798E634D5256464BBBFC3B57852A7F01';
delete from t_level_4 where ipnfsid='0000798E634D5256464BBBFC3B57852A7F01';
delete from t_storageinfo where ipnfsid='0000798E634D5256464BBBFC3B57852A7F01';
delete from t_locationinfo where ipnfsid='0000798E634D5256464BBBFC3B57852A7F01' and itype=0;
delete from t_level_1 where ipnfsid='00000903E09A16D84B6593C81F8A56182AA7';
delete from t_level_4 where ipnfsid='00000903E09A16D84B6593C81F8A56182AA7';
delete from t_storageinfo where ipnfsid='00000903E09A16D84B6593C81F8A56182AA7';
delete from t_locationinfo where ipnfsid='00000903E09A16D84B6593C81F8A56182AA7' and itype=0;
delete from t_level_1 where ipnfsid='000088095A108386418FB3771A93E4C93357';
delete from t_level_4 where ipnfsid='000088095A108386418FB3771A93E4C93357';
delete from t_storageinfo where ipnfsid='000088095A108386418FB3771A93E4C93357';
delete from t_locationinfo where ipnfsid='000088095A108386418FB3771A93E4C93357' and itype=0;
delete from t_level_1 where ipnfsid='0000378765762BDD4924AB08BCA1245A8357';
delete from t_level_4 where ipnfsid='0000378765762BDD4924AB08BCA1245A8357';
delete from t_storageinfo where ipnfsid='0000378765762BDD4924AB08BCA1245A8357';
delete from t_locationinfo where ipnfsid='0000378765762BDD4924AB08BCA1245A8357' and itype=0;
delete from t_level_1 where ipnfsid='00008A6C8CA0A4D144F09F2A8D401F89A40C';
delete from t_level_4 where ipnfsid='00008A6C8CA0A4D144F09F2A8D401F89A40C';
delete from t_storageinfo where ipnfsid='00008A6C8CA0A4D144F09F2A8D401F89A40C';
delete from t_locationinfo where ipnfsid='00008A6C8CA0A4D144F09F2A8D401F89A40C' and itype=0;
delete from t_level_1 where ipnfsid='0000432CAB2848174B9CB45771B1E4DBA544';
delete from t_level_4 where ipnfsid='0000432CAB2848174B9CB45771B1E4DBA544';
delete from t_storageinfo where ipnfsid='0000432CAB2848174B9CB45771B1E4DBA544';
delete from t_locationinfo where ipnfsid='0000432CAB2848174B9CB45771B1E4DBA544' and itype=0;
delete from t_level_1 where ipnfsid='000015F22319051D4E7C9D58590C68FA5446';
delete from t_level_4 where ipnfsid='000015F22319051D4E7C9D58590C68FA5446';
delete from t_storageinfo where ipnfsid='000015F22319051D4E7C9D58590C68FA5446';
delete from t_locationinfo where ipnfsid='000015F22319051D4E7C9D58590C68FA5446' and itype=0;
delete from t_level_1 where ipnfsid='0000FC055F51ECF74F27BA4119147221BE0A';
delete from t_level_4 where ipnfsid='0000FC055F51ECF74F27BA4119147221BE0A';
delete from t_storageinfo where ipnfsid='0000FC055F51ECF74F27BA4119147221BE0A';
delete from t_locationinfo where ipnfsid='0000FC055F51ECF74F27BA4119147221BE0A' and itype=0;
delete from t_level_1 where ipnfsid='000059B5E989EA9E4C019727BA79954B5F4A';
delete from t_level_4 where ipnfsid='000059B5E989EA9E4C019727BA79954B5F4A';
delete from t_storageinfo where ipnfsid='000059B5E989EA9E4C019727BA79954B5F4A';
delete from t_locationinfo where ipnfsid='000059B5E989EA9E4C019727BA79954B5F4A' and itype=0;
delete from t_level_1 where ipnfsid='000005FDACDF0081442DA9E28B713AB515F7';
delete from t_level_4 where ipnfsid='000005FDACDF0081442DA9E28B713AB515F7';
delete from t_storageinfo where ipnfsid='000005FDACDF0081442DA9E28B713AB515F7';
delete from t_locationinfo where ipnfsid='000005FDACDF0081442DA9E28B713AB515F7' and itype=0;
delete from t_level_1 where ipnfsid='000084BEA317D3544F0C953841EFEB258D44';
delete from t_level_4 where ipnfsid='000084BEA317D3544F0C953841EFEB258D44';
delete from t_storageinfo where ipnfsid='000084BEA317D3544F0C953841EFEB258D44';
delete from t_locationinfo where ipnfsid='000084BEA317D3544F0C953841EFEB258D44' and itype=0;
delete from t_level_1 where ipnfsid='000067F9EF42C13C4E18AAC34A5948A2C857';
delete from t_level_4 where ipnfsid='000067F9EF42C13C4E18AAC34A5948A2C857';
delete from t_storageinfo where ipnfsid='000067F9EF42C13C4E18AAC34A5948A2C857';
delete from t_locationinfo where ipnfsid='000067F9EF42C13C4E18AAC34A5948A2C857' and itype=0;
delete from t_level_1 where ipnfsid='0000FBD39840DFF34DFEA332F202D8E07EAC';
delete from t_level_4 where ipnfsid='0000FBD39840DFF34DFEA332F202D8E07EAC';
delete from t_storageinfo where ipnfsid='0000FBD39840DFF34DFEA332F202D8E07EAC';
delete from t_locationinfo where ipnfsid='0000FBD39840DFF34DFEA332F202D8E07EAC' and itype=0;
delete from t_level_1 where ipnfsid='0000A1D22F1C82A340CBA3161CD6981FD4FC';
delete from t_level_4 where ipnfsid='0000A1D22F1C82A340CBA3161CD6981FD4FC';
delete from t_storageinfo where ipnfsid='0000A1D22F1C82A340CBA3161CD6981FD4FC';
delete from t_locationinfo where ipnfsid='0000A1D22F1C82A340CBA3161CD6981FD4FC' and itype=0;
delete from t_level_1 where ipnfsid='0000BE834BEF72444ABDA85CA4633F51634E';
delete from t_level_4 where ipnfsid='0000BE834BEF72444ABDA85CA4633F51634E';
delete from t_storageinfo where ipnfsid='0000BE834BEF72444ABDA85CA4633F51634E';
delete from t_locationinfo where ipnfsid='0000BE834BEF72444ABDA85CA4633F51634E' and itype=0;
delete from t_level_1 where ipnfsid='0000E97D3B635C894B31A5F2F69498B98B8C';
delete from t_level_4 where ipnfsid='0000E97D3B635C894B31A5F2F69498B98B8C';
delete from t_storageinfo where ipnfsid='0000E97D3B635C894B31A5F2F69498B98B8C';
delete from t_locationinfo where ipnfsid='0000E97D3B635C894B31A5F2F69498B98B8C' and itype=0;
delete from t_level_1 where ipnfsid='0000BFD034BB9CE04C1FB6DFCF2CCC3A26F2';
delete from t_level_4 where ipnfsid='0000BFD034BB9CE04C1FB6DFCF2CCC3A26F2';
delete from t_storageinfo where ipnfsid='0000BFD034BB9CE04C1FB6DFCF2CCC3A26F2';
delete from t_locationinfo where ipnfsid='0000BFD034BB9CE04C1FB6DFCF2CCC3A26F2' and itype=0;
delete from t_level_1 where ipnfsid='0000509E43E1B60143E28935BE860B91F664';
delete from t_level_4 where ipnfsid='0000509E43E1B60143E28935BE860B91F664';
delete from t_storageinfo where ipnfsid='0000509E43E1B60143E28935BE860B91F664';
delete from t_locationinfo where ipnfsid='0000509E43E1B60143E28935BE860B91F664' and itype=0;
delete from t_level_1 where ipnfsid='000019E745545F3A4F56A81612303F93FDDA';
delete from t_level_4 where ipnfsid='000019E745545F3A4F56A81612303F93FDDA';
delete from t_storageinfo where ipnfsid='000019E745545F3A4F56A81612303F93FDDA';
delete from t_locationinfo where ipnfsid='000019E745545F3A4F56A81612303F93FDDA' and itype=0;
delete from t_level_1 where ipnfsid='0000EB0C0866953E4453AEC797504BF6358D';
delete from t_level_4 where ipnfsid='0000EB0C0866953E4453AEC797504BF6358D';
delete from t_storageinfo where ipnfsid='0000EB0C0866953E4453AEC797504BF6358D';
delete from t_locationinfo where ipnfsid='0000EB0C0866953E4453AEC797504BF6358D' and itype=0;
delete from t_level_1 where ipnfsid='000019834567F5884982BECFDF2EE23D1EC4';
delete from t_level_4 where ipnfsid='000019834567F5884982BECFDF2EE23D1EC4';
delete from t_storageinfo where ipnfsid='000019834567F5884982BECFDF2EE23D1EC4';
delete from t_locationinfo where ipnfsid='000019834567F5884982BECFDF2EE23D1EC4' and itype=0;
delete from t_level_1 where ipnfsid='00002D59E161AF7243148F13A04AEA13E730';
delete from t_level_4 where ipnfsid='00002D59E161AF7243148F13A04AEA13E730';
delete from t_storageinfo where ipnfsid='00002D59E161AF7243148F13A04AEA13E730';
delete from t_locationinfo where ipnfsid='00002D59E161AF7243148F13A04AEA13E730' and itype=0;
delete from t_level_1 where ipnfsid='000082B34996C86E4480A5CC5ACAFFF95940';
delete from t_level_4 where ipnfsid='000082B34996C86E4480A5CC5ACAFFF95940';
delete from t_storageinfo where ipnfsid='000082B34996C86E4480A5CC5ACAFFF95940';
delete from t_locationinfo where ipnfsid='000082B34996C86E4480A5CC5ACAFFF95940' and itype=0;
delete from t_level_1 where ipnfsid='0000D483C26E08E341468510E989037E5A00';
delete from t_level_4 where ipnfsid='0000D483C26E08E341468510E989037E5A00';
delete from t_storageinfo where ipnfsid='0000D483C26E08E341468510E989037E5A00';
delete from t_locationinfo where ipnfsid='0000D483C26E08E341468510E989037E5A00' and itype=0;
[root@cmschimera HUANGCH]# for i in `cat pnfsid-2015-11-24`

do
psql -U enstore chimera -c "select * from t_inodes where ipnfsid = '$i';"
done

ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+---------------------------+----------------------------
00002FD50ADDA5D94D0780460FB41FEA8E07 | 32768 | 420 | 1 | 9811 | 5063 | 2506243900 | 0 | 2015-11-20 22:10:53.665-06 | 2015-11-20 22:08:52.54-06 | 2015-11-20 22:47:48.352-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+---------------------------+----------------------------
0000EA8FC0046F40498ABCF56C40357F4D95 | 32768 | 420 | 1 | 9811 | 5063 | 3728196831 | 0 | 2015-11-18 12:01:21.166-06 | 2015-11-18 11:53:12.62-06 | 2015-11-18 13:42:24.538-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+-----------+-----+----------------------------+----------------------------+----------------------------
00005C4FF2B3F1724FC9A2A03DEE1BE47869 | 32768 | 420 | 1 | 9811 | 5063 | 264823986 | 0 | 2015-11-18 09:29:26.368-06 | 2015-11-18 09:29:22.671-06 | 2015-11-18 17:43:28.237-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+----------+-----+----------------------------+----------------------------+----------------------------
000063E5C3A5392948B78838F597BA902DCA | 32768 | 420 | 1 | 9811 | 5063 | 59276914 | 0 | 2015-11-21 14:56:47.654-06 | 2015-11-21 14:56:23.699-06 | 2015-11-21 15:10:21.478-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+---------------------------+----------------------------
0000798E634D5256464BBBFC3B57852A7F01 | 32768 | 420 | 1 | 9811 | 5063 | 1061902041 | 0 | 2015-11-18 18:13:37.692-06 | 2015-11-18 18:09:27.87-06 | 2015-11-19 16:15:10.755-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+---------------------------+----------------------------
00000903E09A16D84B6593C81F8A56182AA7 | 32768 | 420 | 1 | 9811 | 5063 | 2208905774 | 0 | 2015-11-20 16:05:04.295-06 | 2015-11-20 16:03:34.11-06 | 2015-11-20 17:29:31.123-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+---------------------------+----------------------------+----------------------------
000088095A108386418FB3771A93E4C93357 | 32768 | 420 | 1 | 9811 | 5063 | 2847438863 | 0 | 2015-11-18 17:18:33.12-06 | 2015-11-18 17:13:51.803-06 | 2015-11-20 05:24:12.407-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+-----------+-----+----------------------------+----------------------------+----------------------------
0000378765762BDD4924AB08BCA1245A8357 | 32768 | 420 | 1 | 9811 | 5063 | 650467464 | 0 | 2015-11-19 14:32:27.395-06 | 2015-11-19 14:30:11.551-06 | 2015-11-19 17:17:37.997-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+----------------------------
00008A6C8CA0A4D144F09F2A8D401F89A40C | 32768 | 420 | 1 | 9811 | 5063 | 4174950725 | 0 | 2015-11-18 05:06:10.576-06 | 2015-11-18 05:05:28.038-06 | 2015-11-18 05:14:25.458-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+---------------------------
0000432CAB2848174B9CB45771B1E4DBA544 | 32768 | 420 | 1 | 9811 | 5063 | 6787146562 | 0 | 2015-11-20 23:33:19.493-06 | 2015-11-20 23:30:54.026-06 | 2015-11-21 01:37:27.03-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+----------+-----+----------------------------+----------------------------+---------------------------
000015F22319051D4E7C9D58590C68FA5446 | 32768 | 420 | 1 | 9811 | 5063 | 35393486 | 0 | 2015-11-20 22:21:23.775-06 | 2015-11-20 22:21:22.796-06 | 2015-11-20 22:32:27.13-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+----------------------------
0000FC055F51ECF74F27BA4119147221BE0A | 32768 | 420 | 1 | 9811 | 5063 | 1551852970 | 0 | 2015-11-19 02:06:16.048-06 | 2015-11-19 02:02:18.802-06 | 2015-11-19 04:45:56.806-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+-----------+-----+----------------------------+----------------------------+----------------------------
000059B5E989EA9E4C019727BA79954B5F4A | 32768 | 420 | 1 | 9811 | 5063 | 307126477 | 0 | 2015-11-18 07:10:43.389-06 | 2015-11-18 07:07:31.169-06 | 2015-11-18 09:50:51.131-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+-----------+-----+----------------------------+----------------------------+----------------------------
000005FDACDF0081442DA9E28B713AB515F7 | 32768 | 420 | 1 | 9811 | 5063 | 781841270 | 0 | 2015-11-19 00:42:20.025-06 | 2015-11-19 00:42:01.489-06 | 2015-11-19 03:51:05.578-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+----------------------------
000084BEA317D3544F0C953841EFEB258D44 | 32768 | 420 | 1 | 9811 | 5063 | 1244848276 | 0 | 2015-11-18 17:51:15.721-06 | 2015-11-18 17:49:27.144-06 | 2015-11-18 20:07:57.998-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+----------+-----+----------------------------+----------------------------+----------------------------
000067F9EF42C13C4E18AAC34A5948A2C857 | 32768 | 420 | 1 | 9811 | 5063 | 35387565 | 0 | 2015-11-20 22:34:18.907-06 | 2015-11-20 22:34:12.363-06 | 2015-11-20 22:53:30.402-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+----------------------------
0000FBD39840DFF34DFEA332F202D8E07EAC | 32768 | 420 | 1 | 9811 | 5063 | 4033889533 | 0 | 2015-11-21 04:39:08.654-06 | 2015-11-21 04:30:35.876-06 | 2015-11-21 05:29:32.756-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+----------+-----+----------------------------+----------------------------+----------------------------
0000A1D22F1C82A340CBA3161CD6981FD4FC | 32768 | 420 | 1 | 9811 | 5063 | 59307605 | 0 | 2015-11-21 15:54:08.248-06 | 2015-11-21 15:54:04.103-06 | 2015-11-21 16:04:29.435-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+----------------------------
0000BE834BEF72444ABDA85CA4633F51634E | 32768 | 420 | 1 | 9811 | 5063 | 1075138794 | 0 | 2015-11-18 18:41:20.133-06 | 2015-11-18 18:39:56.046-06 | 2015-11-19 17:26:00.723-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+----------------------------
0000E97D3B635C894B31A5F2F69498B98B8C | 32768 | 420 | 1 | 9811 | 5063 | 3823022649 | 0 | 2015-11-20 22:32:46.166-06 | 2015-11-20 22:16:48.715-06 | 2015-11-21 01:21:25.339-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+----------+-----+----------------------------+----------------------------+----------------------------
0000BFD034BB9CE04C1FB6DFCF2CCC3A26F2 | 32768 | 420 | 1 | 9811 | 5063 | 35389934 | 0 | 2015-11-20 23:15:16.529-06 | 2015-11-20 23:15:16.079-06 | 2015-11-20 23:29:54.952-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+---------------------------+----------------------------
0000509E43E1B60143E28935BE860B91F664 | 32768 | 420 | 1 | 9811 | 5063 | 4090755874 | 0 | 2015-11-20 07:40:21.133-06 | 2015-11-20 07:38:01.58-06 | 2015-11-20 09:23:57.291-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+----------------------------+----------------------------+----------------------------
000019E745545F3A4F56A81612303F93FDDA | 32768 | 420 | 1 | 9811 | 5063 | 3093371223 | 0 | 2015-11-21 15:50:08.253-06 | 2015-11-21 15:49:26.908-06 | 2015-11-21 15:55:05.982-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+---------------------------+----------------------------+----------------------------
0000EB0C0866953E4453AEC797504BF6358D | 32768 | 420 | 1 | 9811 | 5063 | 1144816280 | 0 | 2015-11-19 13:59:40.81-06 | 2015-11-19 13:59:01.179-06 | 2015-11-19 13:59:01.179-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+----------+-----+---------------------------+----------------------------+----------------------------
000019834567F5884982BECFDF2EE23D1EC4 | 32768 | 420 | 1 | 9811 | 5063 | 35396830 | 0 | 2015-11-21 00:33:19.26-06 | 2015-11-21 00:33:17.675-06 | 2015-11-21 00:56:18.688-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+----------+-----+----------------------------+----------------------------+----------------------------
00002D59E161AF7243148F13A04AEA13E730 | 32768 | 420 | 1 | 9811 | 5063 | 35385739 | 0 | 2015-11-20 22:06:25.748-06 | 2015-11-20 22:05:52.183-06 | 2015-11-20 22:21:21.738-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+------------+-----+---------------------------+----------------------------+----------------------------
000082B34996C86E4480A5CC5ACAFFF95940 | 32768 | 420 | 1 | 9811 | 5063 | 2268334695 | 0 | 2015-11-22 20:10:50.14-06 | 2015-11-22 20:08:50.138-06 | 2015-11-22 21:58:32.404-06
(1 row) ipnfsid | itype | imode | inlink | iuid | igid | isize | iio | ictime | iatime | imtime
--------------------------------------+-------+-------+--------+------+------+-----------+-----+----------------------------+----------------------------+----------------------------
0000D483C26E08E341468510E989037E5A00 | 32768 | 420 | 1 | 9811 | 5063 | 850268677 | 0 | 2015-11-18 17:48:30.735-06 | 2015-11-18 17:48:07.905-06 | 2015-11-20 01:30:43.587-06
(1 row)

root@cmschimera HUANGCH]# psql -U enstore chimera -f cmd1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 0
DELETE 0
DELETE 0
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 0
DELETE 0
DELETE 0
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 0
DELETE 0
DELETE 0
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
DELETE 1
[root@cmschimera HUANGCH]#

#5 Updated by Chih-Hao Huang about 4 years ago

Data strart to flow again. Will check back later.

#6 Updated by Chih-Hao Huang about 4 years ago

  • Status changed from Assigned to Resolved
  • % Done changed from 90 to 100

This has been fixed and is looking fine.
However, there is another kind of error that has files on tape yet still mark precious in the pool.
This is harmless for now and I will open anther task to deal with them next week.
They need to be doubly checked.
They could be in one of the two (or more) conditions:
(1) layer 4 exists and is complete -- only need to make it cached
(2) layer 4 exists yet is empty -- should be treated as in this task.



Also available in: Atom PDF