Project

General

Profile

Bug #11372

HPC: try to optimize the ZFS cleanup process

Added by Gerard Bernabeu Altayo almost 4 years ago. Updated over 3 years ago.

Status:
Resolved
Priority:
Normal
Start date:
01/08/2016
Due date:
% Done:

90%

Estimated time:
Duration:

Description

At PIC I've seen how they're managing the ZFS snapshots and seems like removing incremental snapshots shouldn't be an issue. I should make some tests as it would simplify a few processes.

History

#1 Updated by Gerard Bernabeu Altayo almost 4 years ago

I will experience with nexsandata1 as it's easy to reproduce:

[root@dslustre21 ~]# zfs diff nexsanpool1/rhqbbar@day-3-20160103 nexsanpool1/rhqbbar@day-4-20160104
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/DWF_Prop_exact_4545_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/QLUA_exact_4545_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/DWF_Prop_exact_4565_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/QLUA_exact_4565_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/DWF_Prop_exact_4570_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/QLUA_exact_4570_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/DWF_Prop_exact_4575_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/QLUA_exact_4575_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/DWF_Prop_exact_4590_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/QLUA_exact_4590_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/DWF_Prop_exact_4610_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/QLUA_exact_4610_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/DWF_Prop_exact_4635_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/QLUA_exact_4635_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/DWF_Prop_exact_4650_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/QLUA_exact_4650_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/DWF_Prop_exact_4665_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/QLUA_exact_4665_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/DWF_Prop_exact_4685_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/QLUA_exact_4685_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/DWF_Prop_exact_4700_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/QLUA_exact_4700_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/DWF_Prop_exact_4715_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/QLUA_exact_4715_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/DWF_Prop_exact_4720_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/QLUA_exact_4720_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/DWF_Prop_exact_4725_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/QLUA_exact_4725_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/DWF_Prop_exact_4730_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/QLUA_exact_4730_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/DWF_Prop_exact_4740_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/QLUA_exact_4740_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/DWF_Prop_exact_4750_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/QLUA_exact_4750_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/DWF_Prop_exact_4760_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/QLUA_exact_4760_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/DWF_Prop_exact_4765_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/QLUA_exact_4765_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/DWF_Prop_exact_4770_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/QLUA_exact_4770_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/DWF_Prop_exact_4775_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/QLUA_exact_4775_m0.030_x0_y0_z0_t0.job.sh
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/DWF_Prop_exact_7275_m0.030_x0_y0_z0_t0.qlua
M    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/QLUA_exact_7275_m0.030_x0_y0_z0_t0.job.sh
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/Q24c_4665_m0.030_x0_y0_z0_t0-1351397.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/Q24c_4720_m0.030_x0_y0_z0_t0-1351401.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/Q24c_4725_m0.030_x0_y0_z0_t0-1351402.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/Q24c_4750_m0.030_x0_y0_z0_t0-1351405.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/Q24c_4760_m0.030_x0_y0_z0_t0-1351406.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/Q24c_4770_m0.030_x0_y0_z0_t0-1351408.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/Q24c_4775_m0.030_x0_y0_z0_t0-1351409.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/DWF_Prop_exact_6500_m0.030_x0_y0_z0_t0.qlua
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/QLUA_exact_6500_m0.030_x0_y0_z0_t0.job.sh
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/Q24c_6500_m0.030_x0_y0_z0_t0-1351410.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/Q24c_4570_m0.030_x0_y0_z0_t0-1351391.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/Q24c_4545_m0.030_x0_y0_z0_t0-1351389.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/Q24c_4565_m0.030_x0_y0_z0_t0-1351390.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/Q24c_4575_m0.030_x0_y0_z0_t0-1351392.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/Q24c_4715_m0.030_x0_y0_z0_t0-1351400.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/Q24c_4590_m0.030_x0_y0_z0_t0-1351393.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/Q24c_4610_m0.030_x0_y0_z0_t0-1351394.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/Q24c_4635_m0.030_x0_y0_z0_t0-1351395.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/Q24c_4685_m0.030_x0_y0_z0_t0-1351398.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/Q24c_4700_m0.030_x0_y0_z0_t0-1351399.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/Q24c_4730_m0.030_x0_y0_z0_t0-1351403.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/Q24c_4740_m0.030_x0_y0_z0_t0-1351404.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/Q24c_4765_m0.030_x0_y0_z0_t0-1351407.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/Q24c_4650_m0.030_x0_y0_z0_t0-1351396.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0-1351411.ds2.fnal.gov.out
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/Q24c_4750_m0.030_x0_y0_z0_t0.o1351405
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/Q24c_4665_m0.030_x0_y0_z0_t0.o1351397
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/Q24c_4725_m0.030_x0_y0_z0_t0.o1351402
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/Q24c_4775_m0.030_x0_y0_z0_t0.o1351409
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/Q24c_6500_m0.030_x0_y0_z0_t0.o1351410
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/Q24c_4720_m0.030_x0_y0_z0_t0.o1351401
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/Q24c_4770_m0.030_x0_y0_z0_t0.o1351408
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/Q24c_4545_m0.030_x0_y0_z0_t0.o1351389
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/Q24c_4760_m0.030_x0_y0_z0_t0.o1351406
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/Q24c_4610_m0.030_x0_y0_z0_t0.o1351394
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/Q24c_4565_m0.030_x0_y0_z0_t0.o1351390
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/Q24c_4715_m0.030_x0_y0_z0_t0.o1351400
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/Q24c_4575_m0.030_x0_y0_z0_t0.o1351392
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/Q24c_4730_m0.030_x0_y0_z0_t0.o1351403
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/Q24c_4685_m0.030_x0_y0_z0_t0.o1351398
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/Q24c_4590_m0.030_x0_y0_z0_t0.o1351393
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/Q24c_4700_m0.030_x0_y0_z0_t0.o1351399
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/Q24c_4635_m0.030_x0_y0_z0_t0.o1351395
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/Q24c_4570_m0.030_x0_y0_z0_t0.o1351391
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/Q24c_4740_m0.030_x0_y0_z0_t0.o1351404
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/Q24c_4765_m0.030_x0_y0_z0_t0.o1351407
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/Q24c_4650_m0.030_x0_y0_z0_t0.o1351396
+    /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411
[root@dslustre21 ~]# 
[root@dslustre21 ~]# ll /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411
-rw-r--r-- 1 witzel rhqbbar 13 Jan  4 04:42 /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411
[root@dslustre21 ~]# cat /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411
PBS epilogue
[root@dslustre21 ~]# 

Now I will destroy the dataset where this file was created: nexsanpool1/rhqbbar@day-4-20160104

[root@dslustre21 ~]# zfs list -t snapshot  | grep rhqbbar
nexsanpool1/rhqbbar@day-1-20160101                                 63.2M      -   731G  -
nexsanpool1/rhqbbar@day-2-20160102                                 15.0M      -   731G  -
nexsanpool1/rhqbbar@day-3-20160103                                  432K      -   731G  -
nexsanpool1/rhqbbar@day-4-20160104                                  208K      -   731G  -
nexsanpool1/rhqbbar@day-5-20160105                                 1007K      -   712G  -
nexsanpool1/rhqbbar@day-6-20160106                                  791K      -   713G  -
nexsanpool1/rhqbbar@day-7-20160107                                     0      -   713G  -
nexsanpool2/rhqbbar@day-1-20160101                                 26.8G      -   731G  -
nexsanpool2/rhqbbar@week-1-20160107                                    0      -   713G  -
projectzfspool/projectzfs/rhqbbar@day-1-20160101                   31.6M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-2-20160102                   7.52M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-3-20160103                    216K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-4-20160104                    104K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-5-20160105                    512K      -   801G  -
projectzfspool/projectzfs/rhqbbar@day-6-20160106                    412K      -   801G  -
projectzfspool/projectzfs/rhqbbar@day-7-20160107                   1.01M      -   801G  -
[root@dslustre21 ~]# zfs destroy nexsanpool1/rhqbbar@day-4-20160104
[root@dslustre21 ~]# echo $?
0
[root@dslustre21 ~]# zfs list -t snapshot  | grep rhqbbar
nexsanpool1/rhqbbar@day-1-20160101                                 63.2M      -   731G  -
nexsanpool1/rhqbbar@day-2-20160102                                 15.0M      -   731G  -
nexsanpool1/rhqbbar@day-3-20160103                                  432K      -   731G  -
nexsanpool1/rhqbbar@day-5-20160105                                 1007K      -   712G  -
nexsanpool1/rhqbbar@day-6-20160106                                  791K      -   713G  -
nexsanpool1/rhqbbar@day-7-20160107                                  176K      -   713G  -
nexsanpool2/rhqbbar@day-1-20160101                                 26.8G      -   731G  -
nexsanpool2/rhqbbar@week-1-20160107                                    0      -   713G  -
projectzfspool/projectzfs/rhqbbar@day-1-20160101                   31.6M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-2-20160102                   7.52M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-3-20160103                    216K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-4-20160104                    104K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-5-20160105                    512K      -   801G  -
projectzfspool/projectzfs/rhqbbar@day-6-20160106                    412K      -   801G  -
projectzfspool/projectzfs/rhqbbar@day-7-20160107                   1.01M      -   801G  -
[root@dslustre21 ~]# 

The file still exists, this shows that there is no data loss:

[root@dslustre21 ~]# ll /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411
-rw-r--r-- 1 witzel rhqbbar 13 Jan  4 04:42 /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411
[root@dslustre21 ~]# cat /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411
PBS epilogue
[root@dslustre21 ~]# 

Of course now I'm not capable of restoring the snapshot by applying it because the changes are already in:

[root@dslustre21 ~]# /sbin/zfs send -v -i projectzfspool/projectzfs/rhqbbar@day-3-20160103 projectzfspool/projectzfs/rhqbbar@day-4-20160104 | /sbin/zfs recv nexsanpool1/rhqbbar@day-4-20160104
send from @day-3-20160103 to projectzfspool/projectzfs/rhqbbar@day-4-20160104 estimated size is 1.59M
total estimated size is 1.59M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: destination nexsanpool1/rhqbbar has been modified
since most recent snapshot
warning: cannot send 'projectzfspool/projectzfs/rhqbbar@day-4-20160104': Broken pipe

And forcing it rolls it back:

[root@dslustre21 ~]# /sbin/zfs send -v -i projectzfspool/projectzfs/rhqbbar@day-3-20160103 projectzfspool/projectzfs/rhqbbar@day-4-20160104 | /sbin/zfs recv -F nexsanpool1/rhqbbar@day-4-20160104
send from @day-3-20160103 to projectzfspool/projectzfs/rhqbbar@day-4-20160104 estimated size is 1.59M
total estimated size is 1.59M
TIME        SENT   SNAPSHOT
[root@dslustre21 ~]# zfs list -t snapshot  | grep rhqbbar
nexsanpool1/rhqbbar@day-1-20160101                                 63.2M      -   731G  -
nexsanpool1/rhqbbar@day-2-20160102                                 15.0M      -   731G  -
nexsanpool1/rhqbbar@day-3-20160103                                  432K      -   731G  -
nexsanpool1/rhqbbar@day-4-20160104                                     0      -   731G  -
nexsanpool2/rhqbbar@day-1-20160101                                 26.8G      -   731G  -
nexsanpool2/rhqbbar@week-1-20160107                                    0      -   713G  -
projectzfspool/projectzfs/rhqbbar@day-1-20160101                   31.6M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-2-20160102                   7.52M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-3-20160103                    216K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-4-20160104                    104K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-5-20160105                    512K      -   801G  -
projectzfspool/projectzfs/rhqbbar@day-6-20160106                    412K      -   801G  -
projectzfspool/projectzfs/rhqbbar@day-7-20160107                   1.01M      -   801G  -
[root@dslustre21 ~]# 

Now I will just redo the backups by running the script again:

[root@dslustre21 ~]# echo rhqbbar | /root/zfs-scripts/replica-zfs-snapshot.pl 
Jan 08 10:59:44 (I) Reading dataset list from stdin ..
Jan 08 10:59:44 (I) Making daily snapshots - Snapshot date tag is 8
Jan 08 10:59:44 (I) Replicating snapshots for rhqbbar
Jan 08 10:59:44 (I) Creating daily snapshot: /sbin/zfs send -v -i projectzfspool/projectzfs/rhqbbar@day-4-20160104 projectzfspool/projectzfs/rhqbbar@day-5-20160105 | /sbin/zfs recv -F nexsanpool1/rhqbbar@day-5-20160105
send from @day-4-20160104 to projectzfspool/projectzfs/rhqbbar@day-5-20160105 estimated size is 7.63G
total estimated size is 7.63G
TIME        SENT   SNAPSHOT
10:59:46   57.0K   projectzfspool/projectzfs/rhqbbar@day-5-20160105
10:59:47   26.6M   projectzfspool/projectzfs/rhqbbar@day-5-20160105
10:59:48    102M   projectzfspool/projectzfs/rhqbbar@day-5-20160105
10:59:49    206M   projectzfspool/projectzfs/rhqbbar@day-5-20160105
10:59:50    306M   projectzfspool/projectzfs/rhqbbar@day-5-20160105
...

#2 Updated by Gerard Bernabeu Altayo almost 4 years ago

Conclusion is that the zpool destroy -R is not needed and I can only destroy the snapshots and let the new snapshots come in.

#3 Updated by Gerard Bernabeu Altayo almost 4 years ago

Getting the files to remove for this month:

[root@dslustre21 zfs-scripts]# bash ./delete-gzips.sh > FilesToRemove.20160111.log 2>&1
[root@dslustre21 zfs-scripts]# screen
[detached]

I need to implement the efficient way of doing this (calculate CRC after file generation so that it's in cache!)

#4 Updated by Gerard Bernabeu Altayo almost 4 years ago

The script failed due to SSH permission issues from within the screen:

Permission denied (gssapi-keyex,gssapi-with-mic,keyboard-interactive).

Since I have all the local checksums calculated I will create the final script that processes and removes the files :)

#5 Updated by Gerard Bernabeu Altayo almost 4 years ago

I've created the list:

[root@dslustre21 zfs-scripts]# head /nexsanpool2/filelist-201512.crc
/nexsanpool2/axial-day-1-20160101.gz 364767830
/nexsanpool2/axial-week-1-20160107.gz 3921114051
/nexsanpool2/axial-week-3-20151221.gz 2417836686
/nexsanpool2/axial-week-4-20151228.gz 2621114247
/nexsanpool2/axial-week-5-20151231.gz 1494400073
/nexsanpool2/c51-day-1-20151201.gz 2265708214
/nexsanpool2/c51-day-1-20160101.gz 2900074934
/nexsanpool2/c51-week-1-20151207.gz 3635232688
/nexsanpool2/c51-week-1-20160107.gz 1980908136
/nexsanpool2/c51-week-2-20151214.gz 1045316058

Will now adapt the script to read this file, and then I have to generate them :).

#6 Updated by Gerard Bernabeu Altayo almost 4 years ago

The script to remove the files is ready, I am not uncomenting the final 'rm' because next month I want to launch the rm after careful consideration, but if it works this could be in a cron that runs the 2nd week of the month or so...

[root@dslustre21 zfs-scripts]# bash ./delete-gzips.sh 
There are 63 /nexsanpool2/filelist-201512.crc files to be considered for removal
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-week-3-20151221.gz vs axial-week-3-20151221.gz
INFO: Removing. Checksums are 2417836686 vs 2417836686
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-week-4-20151228.gz vs axial-week-4-20151228.gz
INFO: Removing. Checksums are 2621114247 vs 2621114247
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-week-5-20151231.gz vs axial-week-5-20151231.gz
INFO: Removing. Checksums are 1494400073 vs 1494400073
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-day-1-20151201.gz vs c51-day-1-20151201.gz
INFO: Removing. Checksums are 2265708214 vs 2265708214
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-1-20151207.gz vs c51-week-1-20151207.gz
INFO: Removing. Checksums are 3635232688 vs 3635232688
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-2-20151214.gz vs c51-week-2-20151214.gz
INFO: Removing. Checksums are 1045316058 vs 1045316058
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-3-20151221.gz vs c51-week-3-20151221.gz
INFO: Removing. Checksums are 1954234560 vs 1954234560
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-4-20151228.gz vs c51-week-4-20151228.gz
INFO: Removing. Checksums are 1486962691 vs 1486962691
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-5-20151231.gz vs c51-week-5-20151231.gz
INFO: Removing. Checksums are 1609711399 vs 1609711399
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-day-1-20151201.gz vs charmonium-day-1-20151201.gz
INFO: Removing. Checksums are 1893491997 vs 1893491997
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-1-20151207.gz vs charmonium-week-1-20151207.gz
INFO: Removing. Checksums are 1113410888 vs 1113410888
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-2-20151214.gz vs charmonium-week-2-20151214.gz
    INFO: Removing. Checksums are 1044791556 vs 1044791556
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-3-20151221.gz vs charmonium-week-3-20151221.gz
INFO: Removing. Checksums are 1059013925 vs 1059013925
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-4-20151228.gz vs charmonium-week-4-20151228.gz
INFO: Removing. Checksums are 591282615 vs 591282615
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-5-20151231.gz vs charmonium-week-5-20151231.gz
INFO: Removing. Checksums are 3002811213 vs 3002811213
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-day-1-20151201.gz vs disco-day-1-20151201.gz
INFO: Removing. Checksums are 1917477495 vs 1917477495
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-1-20151207.gz vs disco-week-1-20151207.gz
INFO: Removing. Checksums are 98124554 vs 98124554
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-2-20151214.gz vs disco-week-2-20151214.gz
INFO: Removing. Checksums are 2869313602 vs 2869313602
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-3-20151221.gz vs disco-week-3-20151221.gz
INFO: Removing. Checksums are 688406556 vs 688406556
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-4-20151228.gz vs disco-week-4-20151228.gz
INFO: Removing. Checksums are 3814671458 vs 3814671458
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-5-20151231.gz vs disco-week-5-20151231.gz
INFO: Removing. Checksums are 3636611166 vs 3636611166
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-day-1-20151201.gz vs fourpluseight-day-1-20151201.gz
INFO: Removing. Checksums are 3271725874 vs 3271725874
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-1-20151207.gz vs fourpluseight-week-1-20151207.gz
INFO: Removing. Checksums are 959989706 vs 959989706
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-2-20151214.gz vs fourpluseight-week-2-20151214.gz
INFO: Removing. Checksums are 3852224621 vs 3852224621
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-3-20151221.gz vs fourpluseight-week-3-20151221.gz
INFO: Removing. Checksums are 1408695196 vs 1408695196
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-4-20151228.gz vs fourpluseight-week-4-20151228.gz
INFO: Removing. Checksums are 118311785 vs 118311785
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-5-20151231.gz vs fourpluseight-week-5-20151231.gz
INFO: Removing. Checksums are 1839548120 vs 1839548120
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-day-1-20151201.gz vs hpqcd-day-1-20151201.gz
INFO: Removing. Checksums are 498550980 vs 498550980
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-1-20151207.gz vs hpqcd-week-1-20151207.gz
INFO: Removing. Checksums are 2402566473 vs 2402566473
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-2-20151214.gz vs hpqcd-week-2-20151214.gz
INFO: Removing. Checksums are 1893744883 vs 1893744883
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-3-20151221.gz vs hpqcd-week-3-20151221.gz
INFO: Removing. Checksums are 383154737 vs 383154737
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-4-20151228.gz vs hpqcd-week-4-20151228.gz
INFO: Removing. Checksums are 4135273090 vs 4135273090
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-5-20151231.gz vs hpqcd-week-5-20151231.gz
INFO: Removing. Checksums are 4223353350 vs 4223353350
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-day-1-20151201.gz vs mixbk-day-1-20151201.gz
INFO: Removing. Checksums are 495425356 vs 495425356
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-1-20151207.gz vs mixbk-week-1-20151207.gz
INFO: Removing. Checksums are 4274602064 vs 4274602064
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-2-20151214.gz vs mixbk-week-2-20151214.gz
INFO: Removing. Checksums are 1894860552 vs 1894860552
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-3-20151221.gz vs mixbk-week-3-20151221.gz
INFO: Removing. Checksums are 3871489434 vs 3871489434
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-4-20151228.gz vs mixbk-week-4-20151228.gz
INFO: Removing. Checksums are 1193362265 vs 1193362265
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-5-20151231.gz vs mixbk-week-5-20151231.gz
INFO: Removing. Checksums are 1567245158 vs 1567245158
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-day-1-20151201.gz vs nonperturbativestaggered-day-1-20151201.gz
INFO: Removing. Checksums are 1414143160 vs 1414143160
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-1-20151207.gz vs nonperturbativestaggered-week-1-20151207.gz
INFO: Removing. Checksums are 1555842468 vs 1555842468
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-2-20151214.gz vs nonperturbativestaggered-week-2-20151214.gz
INFO: Removing. Checksums are 2831435499 vs 2831435499
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-3-20151221.gz vs nonperturbativestaggered-week-3-20151221.gz
INFO: Removing. Checksums are 3492496316 vs 3492496316
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-4-20151228.gz vs nonperturbativestaggered-week-4-20151228.gz
INFO: Removing. Checksums are 3638249525 vs 3638249525
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-5-20151231.gz vs nonperturbativestaggered-week-5-20151231.gz
INFO: Removing. Checksums are 1580354178 vs 1580354178
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-day-1-20151201.gz vs ok-action-day-1-20151201.gz
INFO: Removing. Checksums are 3381800064 vs 3381800064
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-1-20151207.gz vs ok-action-week-1-20151207.gz
INFO: Removing. Checksums are 3284813758 vs 3284813758
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-2-20151214.gz vs ok-action-week-2-20151214.gz
INFO: Removing. Checksums are 2853716679 vs 2853716679
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-3-20151221.gz vs ok-action-week-3-20151221.gz
INFO: Removing. Checksums are 2734047468 vs 2734047468
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-4-20151228.gz vs ok-action-week-4-20151228.gz
INFO: Removing. Checksums are 3036693111 vs 3036693111
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-5-20151231.gz vs ok-action-week-5-20151231.gz
INFO: Removing. Checksums are 2246263745 vs 2246263745
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-day-1-20151201.gz vs rbcdwf-day-1-20151201.gz
INFO: Removing. Checksums are 3612825378 vs 3612825378
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-1-20151207.gz vs rbcdwf-week-1-20151207.gz
INFO: Removing. Checksums are 322453349 vs 322453349
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-2-20151214.gz vs rbcdwf-week-2-20151214.gz
INFO: Removing. Checksums are 2662220017 vs 2662220017
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-3-20151221.gz vs rbcdwf-week-3-20151221.gz
INFO: Removing. Checksums are 2247966270 vs 2247966270
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-4-20151228.gz vs rbcdwf-week-4-20151228.gz
INFO: Removing. Checksums are 21250907 vs 21250907
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-5-20151231.gz vs rbcdwf-week-5-20151231.gz
INFO: Removing. Checksums are 1982546892 vs 1982546892
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-day-1-20151201.gz vs rhqbbar-day-1-20151201.gz
INFO: Removing. Checksums are 91887815 vs 91887815
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-1-20151207.gz vs rhqbbar-week-1-20151207.gz
INFO: Removing. Checksums are 3725737770 vs 3725737770
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-2-20151214.gz vs rhqbbar-week-2-20151214.gz
INFO: Removing. Checksums are 1445346914 vs 1445346914
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-3-20151221.gz vs rhqbbar-week-3-20151221.gz
INFO: Removing. Checksums are 3672809767 vs 3672809767
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-4-20151228.gz vs rhqbbar-week-4-20151228.gz
INFO: Removing. Checksums are 1304247893 vs 1304247893
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-5-20151231.gz vs rhqbbar-week-5-20151231.gz
INFO: Removing. Checksums are 2179975191 vs 2179975191
rm -f /nexsanpool2/filelist-201512.crc  /nexsanpool2/axial-week-3-20151221.gz /nexsanpool2/axial-week-4-20151228.gz /nexsanpool2/axial-week-5-20151231.gz /nexsanpool2/c51-day-1-20151201.gz /nexsanpool2/c51-week-1-20151207.gz /nexsanpool2/c51-week-2-20151214.gz /nexsanpool2/c51-week-3-20151221.gz /nexsanpool2/c51-week-4-20151228.gz /nexsanpool2/c51-week-5-20151231.gz /nexsanpool2/charmonium-day-1-20151201.gz /nexsanpool2/charmonium-week-1-20151207.gz /nexsanpool2/charmonium-week-2-20151214.gz /nexsanpool2/charmonium-week-3-20151221.gz /nexsanpool2/charmonium-week-4-20151228.gz /nexsanpool2/charmonium-week-5-20151231.gz /nexsanpool2/disco-day-1-20151201.gz /nexsanpool2/disco-week-1-20151207.gz /nexsanpool2/disco-week-2-20151214.gz /nexsanpool2/disco-week-3-20151221.gz /nexsanpool2/disco-week-4-20151228.gz /nexsanpool2/disco-week-5-20151231.gz /nexsanpool2/fourpluseight-day-1-20151201.gz /nexsanpool2/fourpluseight-week-1-20151207.gz /nexsanpool2/fourpluseight-week-2-20151214.gz /nexsanpool2/fourpluseight-week-3-20151221.gz /nexsanpool2/fourpluseight-week-4-20151228.gz /nexsanpool2/fourpluseight-week-5-20151231.gz /nexsanpool2/hpqcd-day-1-20151201.gz /nexsanpool2/hpqcd-week-1-20151207.gz /nexsanpool2/hpqcd-week-2-20151214.gz /nexsanpool2/hpqcd-week-3-20151221.gz /nexsanpool2/hpqcd-week-4-20151228.gz /nexsanpool2/hpqcd-week-5-20151231.gz /nexsanpool2/mixbk-day-1-20151201.gz /nexsanpool2/mixbk-week-1-20151207.gz /nexsanpool2/mixbk-week-2-20151214.gz /nexsanpool2/mixbk-week-3-20151221.gz /nexsanpool2/mixbk-week-4-20151228.gz /nexsanpool2/mixbk-week-5-20151231.gz /nexsanpool2/nonperturbativestaggered-day-1-20151201.gz /nexsanpool2/nonperturbativestaggered-week-1-20151207.gz /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz /nexsanpool2/nonperturbativestaggered-week-4-20151228.gz /nexsanpool2/nonperturbativestaggered-week-5-20151231.gz /nexsanpool2/ok-action-day-1-20151201.gz /nexsanpool2/ok-action-week-1-20151207.gz /nexsanpool2/ok-action-week-2-20151214.gz /nexsanpool2/ok-action-week-3-20151221.gz /nexsanpool2/ok-action-week-4-20151228.gz /nexsanpool2/ok-action-week-5-20151231.gz /nexsanpool2/rbcdwf-day-1-20151201.gz /nexsanpool2/rbcdwf-week-1-20151207.gz /nexsanpool2/rbcdwf-week-2-20151214.gz /nexsanpool2/rbcdwf-week-3-20151221.gz /nexsanpool2/rbcdwf-week-4-20151228.gz /nexsanpool2/rbcdwf-week-5-20151231.gz /nexsanpool2/rhqbbar-day-1-20151201.gz /nexsanpool2/rhqbbar-week-1-20151207.gz /nexsanpool2/rhqbbar-week-2-20151214.gz /nexsanpool2/rhqbbar-week-3-20151221.gz /nexsanpool2/rhqbbar-week-4-20151228.gz /nexsanpool2/rhqbbar-week-5-20151231.gz
[root@dslustre21 zfs-scripts]# rm -f /nexsanpool2/filelist-201512.crc  /nexsanpool2/axial-week-3-20151221.gz /nexsanpool2/axial-week-4-20151228.gz /nexsanpool2/axial-week-5-20151231.gz /nexsanpool2/c51-day-1-20151201.gz /nexsanpool2/c51-week-1-20151207.gz /nexsanpool2/c51-week-2-20151214.gz /nexsanpool2/c51-week-3-20151221.gz /nexsanpool2/c51-week-4-20151228.gz /nexsanpool2/c51-week-5-20151231.gz /nexsanpool2/charmonium-day-1-20151201.gz /nexsanpool2/charmonium-week-1-20151207.gz /nexsanpool2/charmonium-week-2-20151214.gz /nexsanpool2/charmonium-week-3-20151221.gz /nexsanpool2/charmonium-week-4-20151228.gz /nexsanpool2/charmonium-week-5-20151231.gz /nexsanpool2/disco-day-1-20151201.gz /nexsanpool2/disco-week-1-20151207.gz /nexsanpool2/disco-week-2-20151214.gz /nexsanpool2/disco-week-3-20151221.gz /nexsanpool2/disco-week-4-20151228.gz /nexsanpool2/disco-week-5-20151231.gz /nexsanpool2/fourpluseight-day-1-20151201.gz /nexsanpool2/fourpluseight-week-1-20151207.gz /nexsanpool2/fourpluseight-week-2-20151214.gz /nexsanpool2/fourpluseight-week-3-20151221.gz /nexsanpool2/fourpluseight-week-4-20151228.gz /nexsanpool2/fourpluseight-week-5-20151231.gz /nexsanpool2/hpqcd-day-1-20151201.gz /nexsanpool2/hpqcd-week-1-20151207.gz /nexsanpool2/hpqcd-week-2-20151214.gz /nexsanpool2/hpqcd-week-3-20151221.gz /nexsanpool2/hpqcd-week-4-20151228.gz /nexsanpool2/hpqcd-week-5-20151231.gz /nexsanpool2/mixbk-day-1-20151201.gz /nexsanpool2/mixbk-week-1-20151207.gz /nexsanpool2/mixbk-week-2-20151214.gz /nexsanpool2/mixbk-week-3-20151221.gz /nexsanpool2/mixbk-week-4-20151228.gz /nexsanpool2/mixbk-week-5-20151231.gz /nexsanpool2/nonperturbativestaggered-day-1-20151201.gz /nexsanpool2/nonperturbativestaggered-week-1-20151207.gz /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz /nexsanpool2/nonperturbativestaggered-week-4-20151228.gz /nexsanpool2/nonperturbativestaggered-week-5-20151231.gz /nexsanpool2/ok-action-day-1-20151201.gz /nexsanpool2/ok-action-week-1-20151207.gz /nexsanpool2/ok-action-week-2-20151214.gz /nexsanpool2/ok-action-week-3-20151221.gz /nexsanpool2/ok-action-week-4-20151228.gz /nexsanpool2/ok-action-week-5-20151231.gz /nexsanpool2/rbcdwf-day-1-20151201.gz /nexsanpool2/rbcdwf-week-1-20151207.gz /nexsanpool2/rbcdwf-week-2-20151214.gz /nexsanpool2/rbcdwf-week-3-20151221.gz /nexsanpool2/rbcdwf-week-4-20151228.gz /nexsanpool2/rbcdwf-week-5-20151231.gz /nexsanpool2/rhqbbar-day-1-20151201.gz /nexsanpool2/rhqbbar-week-1-20151207.gz /nexsanpool2/rhqbbar-week-2-20151214.gz /nexsanpool2/rhqbbar-week-3-20151221.gz /nexsanpool2/rhqbbar-week-4-20151228.gz /nexsanpool2/rhqbbar-week-5-20151231.gz

#7 Updated by Gerard Bernabeu Altayo almost 4 years ago

ecrc creation automated with the gzip creation:

[root@dslustre21 nexsanpool2]# echo c51 | /root/zfs-scripts/replica-zfs-snapshot.pl 
Jan 12 17:37:33 (I) Reading dataset list from stdin ..
Jan 12 17:37:33 (I) Making daily snapshots - Snapshot date tag is 12
Jan 12 17:37:33 (I) Replicating snapshots for c51
Jan 12 17:37:33 (I) Making weekly snapshots
Jan 12 17:37:33 (I) Checking weekly snapshots for c51
Jan 12 17:37:33 (I) There should exist week-1-20160107 . Need to create 
Jan 12 17:37:33 (I) Making GZIPs
Jan 12 17:37:33 (I) Making GZIP for c51
Jan 12 17:37:33 (I) /nexsanpool2/c51-day-1-20160101.gz already exists
Jan 12 17:37:33 (I) Creating /nexsanpool2/c51-week-1-20160107.gz with: (/sbin/zfs send -i nexsanpool2/c51@day-1-20160101 nexsanpool2/c51@week-1-20160107 | gzip > /nexsanpool2/c51-week-1-20160107.gz && (echo -n 'c51-week-1-20160107.gz ' && ecrc /nexsanpool2/c51-week-1-20160107.gz | cut -d' ' -f 2 ) >> /nexsanpool2/filelist-201601.crc )&

Now what's missing is to fully automate the copies to tape and gzip removal.

#8 Updated by Gerard Bernabeu Altayo almost 4 years ago

In order to run the log rotation script I should use a line like the following (from the monthly cron):

 /root/zfs-scripts/delete-gzips.sh >> /root/zfs-scripts/delete-zfs-gzips.log 2>&1

Adding monitoring for this error logs:

[root@dslustre21 check-mk-agent]# diff logwatch.cfg ../logwatch.cfg 
58a59,64
> 
> #Checks for ZFS backups
> /root/zfs-scripts/*zfs*.log
>  C  (E) 
> 
> 
[root@dslustre21 check-mk-agent]# mv ../logwatch.cfg .
mv: overwrite `./logwatch.cfg'? yes
[root@dslustre21 check-mk-agent]# pwd
/etc/check-mk-agent
[root@dslustre21 check-mk-agent]# ll /etc/check-mk-agent/logwatch.cfg 
-rw-r--r-- 1 root root 2456 Jan 13 10:16 /etc/check-mk-agent/logwatch.cfg

Then I added the new 4 services via checkmk's WATO

#9 Updated by Gerard Bernabeu Altayo almost 4 years ago

There is clearly a bug for my script, the CRC file looks like this:

[root@dslustre21 ~]# cat /nexsanpool2/filelist-201601.crc
axial-day-1-20160101.gz 364767830
axial-week-1-20160107.gz 3921114051
c51-day-1-20160101.gz 2900074934
charmonium-day-1-20160101.gz 2396880006
charmonium-week-1-20160107.gz 609896209
disco-day-1-20160101.gz 2245895316
disco-week-1-20160107.gz 1154236981
fourpluseight-day-1-20160101.gz 770351881
fourpluseight-week-1-20160107.gz 3778889209
heavylight-day-1-20160101.gz 1382710575
heavylight-week-1-20160107.gz 1064513988
hpqcd-day-1-20160101.gz 3915399009
hpqcd-week-1-20160107.gz 1012015823
mixbk-day-1-20160101.gz 860111476
mixbk-week-1-20160107.gz 3952952712
nonperturbativestaggered-day-1-20160101.gz 299362089
nonperturbativestaggered-week-1-20160107.gz 4190391641
ok-action-day-1-20160101.gz 3535741520
ok-action-week-1-20160107.gz 3533849934
rbcdwf-day-1-20160101.gz 3077220975
rbcdwf-week-1-20160107.gz 2595766803
rhqbbar-day-1-20160101.gz 2816352350
rhqbbar-week-1-20160107.gz 2393381739
c51-week-1-20160107.gz 259932587
charmonium-week-2-20160114.gz c51-week-2-20160114.gz disco-week-2-20160114.gz fourpluseight-week-2-20160114.gz mixbk-week-2-20160114.gz nonperturbativestaggered-week-2-20160114.gz rbcdwf-week-2-20160114.gz ok-action-week-2-20160114.gz hpqcd-week-2-20160114.gz axial-week-2-20160114.gz rhqbbar-week-2-20160114.gz heavylight-week-2-20160114.gz [root@dslustre21 ~]# 

ie: it is missing all the CRCs, and new lines...

At least no new error was logged and all the snapshots happened as expected:

[root@dslustre21 zfs-scripts]# grep '(E)' *zfs*.log | grep 'Jan 1'
daily-zfs-replica-snapshot.log:Jan 10 02:00:02 (E) Failed (256) to execute: /sbin/zfs send -v -i nexsanpool1/axial@day-1-20160101 nexsanpool1/axial@day-7-20160107 | /sbin/zfs recv nexsanpool2/axial@week-1-20160107
daily-zfs-replica-snapshot.log:Jan 11 02:00:45 (E) Failed (256) to execute: /sbin/zfs send -v -i nexsanpool1/axial@day-1-20160101 nexsanpool1/axial@day-7-20160107 | /sbin/zfs recv nexsanpool2/axial@week-1-20160107
delete-zfs-gzips.log:Wed Jan 13 10:04:35 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING
delete-zfs-gzips.log:Wed Jan 13 10:26:20 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING
delete-zfs-gzips.log:Wed Jan 13 10:29:16 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING
delete-zfs-gzips.log:Wed Jan 13 10:38:27 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING
delete-zfs-gzips.log:Wed Jan 13 10:39:26 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING
delete-zfs-gzips.log:Wed Jan 13 10:39:46 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING
delete-zfs-gzips.log:Wed Jan 13 11:18:46 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING
[root@dslustre21 zfs-scripts]# 

If I look at the logs, I see:

[root@dslustre21 zfs-scripts]# tail  daily-zfs-replica-snapshot.log
Jan 15 02:05:03 (I) Making GZIP for rhqbbar
Jan 15 02:05:03 (I) /nexsanpool2/rhqbbar-day-1-20160101.gz already exists
Jan 15 02:05:03 (I) /nexsanpool2/rhqbbar-week-1-20160107.gz already exists
sh: ecrc: command not found
Jan 15 02:05:03 (I) Creating /nexsanpool2/rhqbbar-week-2-20160114.gz with: (/sbin/zfs send -i nexsanpool2/rhqbbar@week-1-20160107 nexsanpool2/rhqbbar@week-2-20160114 | gzip > /nexsanpool2/rhqbbar-week-2-20160114.gz && (echo -n 'rhqbbar-week-2-20160114.gz ' && ecrc /nexsanpool2/rhqbbar-week-2-20160114.gz | cut -d' ' -f 2 ) >> /nexsanpool2/filelist-201601.crc )&
sh: ecrc: command not found
sh: ecrc: command not found
sh: ecrc: command not found
sh: ecrc: command not found
sh: ecrc: command not found
[root@dslustre21 zfs-scripts]# 

Soo, I need to add the full path for ecrc and this will probably work!

#10 Updated by Gerard Bernabeu Altayo over 3 years ago

Done, it is now working fine:

Jan 19 10:42:40 (I) Making GZIP for rhqbbar
Jan 19 10:42:40 (I) /nexsanpool2/rhqbbar-day-1-20160101.gz already exists
Jan 19 10:42:40 (I) /nexsanpool2/rhqbbar-week-1-20160107.gz already exists
Jan 19 10:42:40 (I) Creating /nexsanpool2/rhqbbar-week-2-20160114.gz in the background with: (/sbin/zfs send -i nexsanpool2/rhqbbar@week-1-20160107 nexsanpool2/rhqbbar@week-2-20160114 | /bin/gzip > /nexsanpool2/rhqbbar-week-2-20160114.gz && (echo -n 'rhqbbar-week-2-20160114.gz ' && /usr/local/bin/ecrc /nexsanpool2/rhqbbar-week-2-20160114.gz | cut -d' ' -f 2 ) >> /nexsanpool2/filelist-201601.crc )&
[root@dslustre21 zfs-scripts]# cat /nexsanpool2/filelist-201601.crc 
axial-day-1-20160101.gz 364767830
axial-week-1-20160107.gz 3921114051
c51-day-1-20160101.gz 2900074934
charmonium-day-1-20160101.gz 2396880006
charmonium-week-1-20160107.gz 609896209
disco-day-1-20160101.gz 2245895316
disco-week-1-20160107.gz 1154236981
fourpluseight-day-1-20160101.gz 770351881
fourpluseight-week-1-20160107.gz 3778889209
heavylight-day-1-20160101.gz 1382710575
heavylight-week-1-20160107.gz 1064513988
hpqcd-day-1-20160101.gz 3915399009
hpqcd-week-1-20160107.gz 1012015823
mixbk-day-1-20160101.gz 860111476
mixbk-week-1-20160107.gz 3952952712
nonperturbativestaggered-day-1-20160101.gz 299362089
nonperturbativestaggered-week-1-20160107.gz 4190391641
ok-action-day-1-20160101.gz 3535741520
ok-action-week-1-20160107.gz 3533849934
rbcdwf-day-1-20160101.gz 3077220975
rbcdwf-week-1-20160107.gz 2595766803
rhqbbar-day-1-20160101.gz 2816352350
rhqbbar-week-1-20160107.gz 2393381739
c51-week-1-20160107.gz 259932587
charmonium-week-2-20160114.gz 218384547
c51-week-2-20160114.gz 2372418994
disco-week-2-20160114.gz 4010689085
fourpluseight-week-2-20160114.gz 1536640057
mixbk-week-2-20160114.gz 3358277476
nonperturbativestaggered-week-2-20160114.gz 2586593235
ok-action-week-2-20160114.gz 2669561960
rbcdwf-week-2-20160114.gz 115425591
[root@dslustre21 zfs-scripts]# 

Also fixed the watchlog.cfg config to escape the (, now it looks like:

#Checks for ZFS backups
/root/zfs-scripts/*zfs*.log
 C  \(E\) 

#11 Updated by Gerard Bernabeu Altayo over 3 years ago

The backup process still has some bug, in this case a race condition that causes the CRC to be written wrong:

[root@dslustre21 ~]# tail -12 /nexsanpool2/filelist-201601.crc 
c51-week-4-20160128.gz charmonium-week-4-20160128.gz 1189101752
2019846639
disco-week-4-20160128.gz 2641314519
fourpluseight-week-4-20160128.gz 149965703
hpqcd-week-4-20160128.gz 1946370591
mixbk-week-4-20160128.gz 2044020069
ok-action-week-4-20160128.gz 3095743034
rbcdwf-week-4-20160128.gz 336871318
axial-week-4-20160128.gz 3756750631
rhqbbar-week-4-20160128.gz 604327362
heavylight-week-4-20160128.gz 2042313975
nonperturbativestaggered-week-4-20160128.gz 1802902312
[root@dslustre21 ~]# 
[root@dslustre21 zfs-scripts]# git diff
diff --git a/replica-zfs-snapshot.pl b/replica-zfs-snapshot.pl
index 32f9103..38a7308 100755
--- a/replica-zfs-snapshot.pl
+++ b/replica-zfs-snapshot.pl
@@ -147,9 +147,9 @@ foreach my $dataset (@datasets) {
       my $date=(split "-", $nexan2snaps[$i])[2];
       my $crcfile="/nexsanpool2/filelist-".(substr $date, 0, 6).".crc";
       if (grep /day-1-.+/, $nexan2snaps[$i]) {
-        $cmd="($ZFS send nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i].gz ' && /usr/local/bin/ecrc $dest | cut -d' ' -f 2) >> $c
+        $cmd="(/bin/sleep 1; $ZFS send nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i].gz ' && /usr/local/bin/ecrc $dest | cut -d'
       } else {
-        $cmd="($ZFS send -i nexsanpool2/$dataset\@$nexan2snaps[$i-1] nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i].gz ' && /usr/
+        $cmd="(/bin/sleep 1; $ZFS send -i nexsanpool2/$dataset\@$nexan2snaps[$i-1] nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i]
       }
       &printLog("I","Creating $dest in the background with: $cmd");
       system("$cmd");
[root@dslustre21 zfs-scripts]# git commit -m'adding sleep to minimize the race condition chances' -a
[master e7eb9bd] adding sleep to minimize the race condition chances

For now I will NOT fix the file as I want the checks to complain that something is wrong at the end of the month (in 3 days)

#12 Updated by Gerard Bernabeu Altayo over 3 years ago

Fixed the logic, now it works!

[root@dslustre21 ~]# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
nexsanpool1 16.2T 7.74T 8.51T 47% 1.00x ONLINE -
nexsanpool2 16.2T 13.3T 2.98T 81% 1.00x ONLINE -
projectzfspool 7.25T 5.56T 1.69T 76% 1.00x ONLINE -
[root@dslustre21 ~]# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
nexsanpool1/axial@day-1-20160201 0 - 13.1G -
nexsanpool1/c51@day-1-20160201 0 - 1.04G -
nexsanpool1/charmonium@day-1-20160201 0 - 615K -
nexsanpool1/disco@day-1-20160201 0 - 6.29G -
nexsanpool1/fourpluseight@day-1-20160201 0 - 22.1G -
nexsanpool1/heavylight@day-1-20160201 0 - 2.89T -
nexsanpool1/hpqcd@day-1-20160201 0 - 778G -
nexsanpool1/mixbk@day-1-20160201 0 - 268G -
nexsanpool1/nonperturbativestaggered@day-1-20160201 0 - 70.1G -
nexsanpool1/ok-action@day-1-20160201 0 - 8.79G -
nexsanpool1/rbcdwf@day-1-20160201 0 - 499G -
nexsanpool1/rhqbbar@day-1-20160201 0 - 654G -
nexsanpool2/axial@day-1-20160201 0 - 13.1G -
nexsanpool2/c51@day-1-20160201 0 - 1.04G -
nexsanpool2/charmonium@day-1-20160201 0 - 615K -
nexsanpool2/disco@day-1-20160201 0 - 6.29G -
nexsanpool2/fourpluseight@day-1-20160201 0 - 22.1G -
nexsanpool2/heavylight@day-1-20160201 0 - 2.89T -
nexsanpool2/hpqcd@day-1-20160201 0 - 778G -
nexsanpool2/mixbk@day-1-20160201 0 - 268G -
nexsanpool2/nonperturbativestaggered@day-1-20160201 0 - 70.1G -
nexsanpool2/ok-action@day-1-20160201 0 - 8.79G -
nexsanpool2/rbcdwf@day-1-20160201 0 - 499G -
nexsanpool2/rhqbbar@day-1-20160201 0 - 654G -
projectzfspool/projectzfs/axial@day-1-20160201 4.45M - 12.9G -
projectzfspool/projectzfs/c51@day-1-20160201 0 - 826M -
projectzfspool/projectzfs/charmonium@day-1-20160201 220K - 440K -
projectzfspool/projectzfs/disco@day-1-20160201 0 - 10.7G -
projectzfspool/projectzfs/fourpluseight@day-1-20160201 0 - 26.5G -
projectzfspool/projectzfs/heavylight@day-1-20160201 4.39M - 3.14T -
projectzfspool/projectzfs/hpqcd@day-1-20160201 0 - 506G -
projectzfspool/projectzfs/mixbk@day-1-20160201 0 - 487G -
projectzfspool/projectzfs/nonperturbativestaggered@day-1-20160201 0 - 152G -
projectzfspool/projectzfs/ok-action@day-1-20160201 0 - 12.6G -
projectzfspool/projectzfs/rbcdwf@day-1-20160201 0 - 572G -
projectzfspool/projectzfs/rhqbbar@day-1-20160201 0 - 694G -
[root@dslustre21 ~]# cd /root/zfs-scripts/
[root@dslustre21 zfs-scripts]# git log | head
commit bb56b57b5f071dda28bae35158f900011edb14f2
Author: root <>
Date: Tue Feb 2 14:40:43 2016 -0600

fixed a few bugs

commit e7eb9bdde8683f05bb5d59f08bb8b53b174b9005
Author: root <>
Date: Fri Jan 29 13:53:49 2016 -0600

[root@dslustre21 zfs-scripts]# git show bb56b57b5f071dda28bae35158f900011edb14f2
commit bb56b57b5f071dda28bae35158f900011edb14f2
Author: root <>
Date: Tue Feb 2 14:40:43 2016 -0600

fixed a few bugs

diff --git a/first-of-month.pl b/first-of-month.pl
index 7d43a26..404f86c 100755
--- a/first-of-month.pl
+++ b/first-of-month.pl
@ -2,9 +2,9 @ # # This script is meant to run on the 2nd of the month to initialize backups for a next month, once all gzips have been generated and the gzips are good: # 1. Continue only if last_month_day-1 and current_month_day-1 exist in projectzfs and all GZIPs (5 per experiment) are there
# 2. Destroy all zpools&replicas from nexsanpool1 and nexsanpool2
# 3. Copy day-1 from projectzfs to nexsanpool1 and from nexsanpool1 to nexsanpool2
-# 4. Remove all last month's snapshots from projecdtzfs
# 2. Copy day-1 incremental from projectzfs to nexsanpool1
# 3. Copy day-1 incremental from nexsanpool1 to nexsanpool2
+# 4. Remove all last month's snapshots from projecdtzfs, nexsanpool1 and nexsanpool2

use Date::Manip;

@ -35,8 +35,10 @ while(<>)
}

-my @datasetstodestroy;
+my @datasetstoreplicate;

# 1. Continue only if last_month_day-1 and current_month_day-1 exist in projectzfs and all GZIPs (5 per experiment) are there
# 2. Copy day-1 incremental from projectzfs to nexsanpool1
DATASET: foreach $dataset (datasets) {
&printLog("I","Checking snapshots for $dataset");
my @projectzfspoolsnaps=split "\n", `$ZFS list -H -r -t snapshot -o name projectzfspool/projectzfs/$dataset | cut -d
-f 2`;
@ -70,37 +72,17 @ DATASET: foreach $dataset (@datasets) {
}

if ( (scalar missingsnaps == 0) && grep(/day-1-$this_month_day1/,@projectzfspoolsnaps) && grep(/day-1-$last_month_day1/,@projectzfspoolsnaps) ) {
- # 2. Destroy all dataset snapshots from nexsanpool1
- my @nexan1snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool1/$dataset | cut -d
f 2`;
foreach my $snap (nexsan1snaps) {
- $cmd="$ZFS destroy -v nexsanpool1/$dataset\
$snap";
- &printLog("I",$cmd);
- system($cmd);
- if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n");
- }
- } # 3. Create the first snapshot of the month: day-1 from projectzfs to nexsanpool1. This used to be a big simple send!
$cmd="$ZFS send v -i projectzfspool/projectzfs/$dataset\day-".UnixDate($last_month_last_day, "%d")."-".UnixDate($last_month_last_day, "%Y%m%d")." projectzfspool/projectzfs/$d
&printLog("I",$cmd);
system($cmd);
if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n NOT cleaning up nexsanpool2/$dataset\n nor removing projectzfspool/projectzfs/$dataset");
+ printLog ("E","Failed ($?) to execute: $cmd\n NOT cleaning up nexsanpool(1|2)/$dataset\n nor removing projectzfspool/projectzfs/$dataset");
next DATASET;
}
else {
- #If we succeed we add to the list of datasets to destroy from nexsanpool2
- push @datasetstodestroy, $dataset;
- }
- # 4. Remove all last month's snapshots from projecdtzfs
- @toremove=grep(/day-.+-$last_month.+/, @projectzfspoolsnaps);
- foreach my $snap (@toremove) {
- $cmd="$ZFS destroy -v projectzfspool/projectzfs/$dataset\
$snap";
&printLog("I",$cmd);
- system($cmd);
- if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n");
- }
+ #If we succeed we add to the list of datasets to destroy from projectzfs and nexsanpool2
+ push @datasetstoreplicate, $dataset;
}
} else {
@ -109,12 +91,32 @ DATASET: foreach $dataset (@datasets) {
}

+# 3. Copy day-1 incremental from nexsanpool1 to nexsanpool2
+my @datasetstodestroy;
+foreach $dataset (@datasetstoreplicate) {

+ #Copy from nexsanpool1 to nexsanpool2
+ $cmd="$ZFS send v -i nexsanpool1/$dataset\@day".UnixDate($last_month_last_day, "%d")."-".UnixDate($last_month_last_day, "%Y%m%d")." nexsanpool1/$dataset\@day-1-$this_month_da
+ #$cmd="$ZFS send -v nexsanpool1/$dataset\@day-1-$this_month_day1 | $ZFS recv nexsanpool2/$dataset &";
+ &printLog("I",$cmd);
+ system($cmd);
+ if ($? != 0) {
+ printLog ("E","Failed ($?) to execute: $cmd\n NOT cleaning up nexsanpool(1|2)/$dataset\n nor removing projectzfspool/projectzfs/$dataset");
+ next DATASET;
+ }
+ else {
+ #If we succeed we add to the list of datasets to destroy from projectzfs and nexsanpool2
+ push @datasetstodestroy, $dataset;
+ }
+}

+# 4. Remove all last month's snapshots from projecdtzfs, nexsanpool1 and nexsanpool2
foreach $dataset (@datasetstodestroy) {

- my nexan2snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool2/$dataset | cut -d f 2`;
foreach my $snap (nexsan2snaps) {
+ # Destroy all last month's dataset snapshots from nexsanpool2
+ my @nexsan2snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool2/$dataset | cut -d
-f 2`;
+ toremove=grep(/.+-$last_month.+/, @nexsan2snaps);
+ foreach my $snap (@toremove) {
$cmd="$ZFS destroy -v nexsanpool2/$dataset\
$snap";
&printLog("I",$cmd);
system($cmd);
@ -123,13 +125,28 @ foreach $dataset (@datasetstodestroy) {
}
}

- #Copy from nexsanpool1 to nexsanpool2
- $cmd="$ZFS send -v -i projectzfspool/projectzfs/$dataset\day-".UnixDate($last_month_last_day, "%d")."-".UnixDate($last_month_last_day, "%Y%m%d")." projectzfspool/projectzfs/$d
- #$cmd="$ZFS send -v nexsanpool1/$dataset\@day-1-$this_month_day1 | $ZFS recv nexsanpool2/$dataset &";
- &printLog("I",$cmd);
- system($cmd);
- if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n");
+ # Destroy all last month's dataset snapshots from nexsanpool1
+ my @nexsan1snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool1/$dataset | cut -d
-f 2`;
+ toremove=grep(/day-.+-$last_month.+/, @nexsan1snaps);
+ foreach my $snap (@toremove) {
+ $cmd="$ZFS destroy -v nexsanpool1/$dataset\
$snap";
+ &printLog("I",$cmd);
+ system($cmd);
+ if ($? != 0) {
+ printLog ("E","Failed ($?) to execute: $cmd\n");
+ }
+ }

# Remove all last month's snapshots from projectzfs
+ my projectzfspoolsnaps=split "\n", `$ZFS list -H -r -t snapshot -o name projectzfspool/projectzfs/$dataset | cut -d -f 2`;
+ toremove=grep(/day-.+-$last_month.+/, @projectzfspoolsnaps);
+ foreach my $snap (@toremove) {
+ $cmd="$ZFS destroy -v projectzfspool/projectzfs/$dataset\
$snap";
+ &printLog("I",$cmd);
+ system($cmd);
+ if ($? != 0) {
+ printLog ("E","Failed ($?) to execute: $cmd\n");
+ }
}

}
[root@dslustre21 zfs-scripts]#

#13 Updated by Gerard Bernabeu Altayo over 3 years ago

  • % Done changed from 0 to 90

Now the gzip removal does not work because it sees that the files are not on tape yet:

[root@dslustre21 ~]# /root/zfs-scripts/delete-gzips.sh
ls: /pnfs/lqcd/backups/lqcdproj_backups/project_backup_*/*/filelist-201601.crc: No such file or directory
dirname: missing operand
Try `dirname --help' for more information.
Tue Feb  2 17:14:17 CST 2016 (E) Could not locate a complete filelist-*.crc on ENSTORE. STOPPING
[root@dslustre21 ~]# echo $?
98
[root@dslustre21 ~]# 

I am adding the cleanup script in a crontab entry, to run on the 4th from dslustre21:


#Monthtly GZIP cleanup. Runs on the 4th of the month to make sure all GZIPs have been removed, only works if tape copies succeeded and after checking checksum is OK. Copy to tape is
 not automated here! Copy to tape runs from [root@lqcdsrm ~]# /home/gerard1/backups_to_tape.sh. 
00 12 4 * * (/root/zfs-scripts/delete-gzips.sh >> /root/zfs-scripts/delete-zfs-gzips.log 2>&1)

Also now automatically running the copy to tape from lqcdsrm crontab now:

# Montthly copy of the GZIPs to tape, GZIP cleanup done from dslustre21
00 16 1 * * (/home/gerard1/backups_to_tape.sh >> /home/gerard1/backups_to_tape.log 2>&1)

I am running it manually now (from a screen), and next month we'll see if it works! If it does, I should tweak the script to have 'standard' greppable output and add a checkmk sensor for it...

#14 Updated by Gerard Bernabeu Altayo over 3 years ago

All files where transferred successfully to tape:

[root@lqcdsrm ~]# grep 'Overall rate' /var/log/lqcd-backup/encp_bkp.2016-02-02--172433.log | wc -l
73

Tomorrow the script that does the removals will do all necessary checks and remove the files, I tried with just an echo on the RM and it worked fine, so I removed the echo:

[root@dslustre21 ~]# /root/zfs-scripts/delete-gzips.sh
Wed Feb  3 12:06:50 CST 2016 (I) There are 72 /nexsanpool2/filelist-201601.crc files to be considered for removal
Wed Feb  3 12:06:50 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-day-1-20160101.gz vs axial-day-1-20160101.gz
Wed Feb  3 12:06:52 CST 2016 (I) Removing. Checksums are 364767830 vs 364767830
Wed Feb  3 12:06:52 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-1-20160107.gz vs axial-week-1-20160107.gz
Wed Feb  3 12:06:53 CST 2016 (I) Removing. Checksums are 3921114051 vs 3921114051
Wed Feb  3 12:06:53 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-day-1-20160101.gz vs c51-day-1-20160101.gz
Wed Feb  3 12:06:53 CST 2016 (I) Removing. Checksums are 2900074934 vs 2900074934
Wed Feb  3 12:06:53 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-day-1-20160101.gz vs charmonium-day-1-20160101.gz
Wed Feb  3 12:06:53 CST 2016 (I) Removing. Checksums are 2396880006 vs 2396880006
Wed Feb  3 12:06:53 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-1-20160107.gz vs charmonium-week-1-20160107.gz
Wed Feb  3 12:06:54 CST 2016 (I) Removing. Checksums are 609896209 vs 609896209
Wed Feb  3 12:06:54 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-day-1-20160101.gz vs disco-day-1-20160101.gz
Wed Feb  3 12:06:54 CST 2016 (I) Removing. Checksums are 2245895316 vs 2245895316
Wed Feb  3 12:06:54 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-1-20160107.gz vs disco-week-1-20160107.gz
Wed Feb  3 12:06:54 CST 2016 (I) Removing. Checksums are 1154236981 vs 1154236981
Wed Feb  3 12:06:54 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-day-1-20160101.gz vs fourpluseight-day-1-20160101.gz
Wed Feb  3 12:06:55 CST 2016 (I) Removing. Checksums are 770351881 vs 770351881
Wed Feb  3 12:06:55 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-1-20160107.gz vs fourpluseight-week-1-20160107.gz
Wed Feb  3 12:06:55 CST 2016 (I) Removing. Checksums are 3778889209 vs 3778889209
Wed Feb  3 12:06:55 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-day-1-20160101.gz vs heavylight-day-1-20160101.gz
Wed Feb  3 12:06:55 CST 2016 (I) Removing. Checksums are 1382710575 vs 1382710575
Wed Feb  3 12:06:55 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-1-20160107.gz vs heavylight-week-1-20160107.gz
Wed Feb  3 12:06:56 CST 2016 (I) Removing. Checksums are 1064513988 vs 1064513988
Wed Feb  3 12:06:56 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-day-1-20160101.gz vs hpqcd-day-1-20160101.gz
Wed Feb  3 12:06:56 CST 2016 (I) Removing. Checksums are 3915399009 vs 3915399009
Wed Feb  3 12:06:56 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-1-20160107.gz vs hpqcd-week-1-20160107.gz
Wed Feb  3 12:06:56 CST 2016 (I) Removing. Checksums are 1012015823 vs 1012015823
Wed Feb  3 12:06:56 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-day-1-20160101.gz vs mixbk-day-1-20160101.gz
Wed Feb  3 12:06:57 CST 2016 (I) Removing. Checksums are 860111476 vs 860111476
Wed Feb  3 12:06:57 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-1-20160107.gz vs mixbk-week-1-20160107.gz
Wed Feb  3 12:06:57 CST 2016 (I) Removing. Checksums are 3952952712 vs 3952952712
Wed Feb  3 12:06:57 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-day-1-20160101.gz vs nonperturbativestaggered-day-1-20160101.gz
Wed Feb  3 12:06:57 CST 2016 (I) Removing. Checksums are 299362089 vs 299362089
Wed Feb  3 12:06:57 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-1-20160107.gz vs nonperturbativestaggered-week-1-20160107.gz
Wed Feb  3 12:06:58 CST 2016 (I) Removing. Checksums are 4190391641 vs 4190391641
Wed Feb  3 12:06:58 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-day-1-20160101.gz vs ok-action-day-1-20160101.gz
Wed Feb  3 12:06:58 CST 2016 (I) Removing. Checksums are 3535741520 vs 3535741520
Wed Feb  3 12:06:58 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-1-20160107.gz vs ok-action-week-1-20160107.gz
Wed Feb  3 12:06:58 CST 2016 (I) Removing. Checksums are 3533849934 vs 3533849934
Wed Feb  3 12:06:58 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-day-1-20160101.gz vs rbcdwf-day-1-20160101.gz
Wed Feb  3 12:06:59 CST 2016 (I) Removing. Checksums are 3077220975 vs 3077220975
Wed Feb  3 12:06:59 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-1-20160107.gz vs rbcdwf-week-1-20160107.gz
Wed Feb  3 12:06:59 CST 2016 (I) Removing. Checksums are 2595766803 vs 2595766803
Wed Feb  3 12:06:59 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-day-1-20160101.gz vs rhqbbar-day-1-20160101.gz
Wed Feb  3 12:06:59 CST 2016 (I) Removing. Checksums are 2816352350 vs 2816352350
Wed Feb  3 12:06:59 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-1-20160107.gz vs rhqbbar-week-1-20160107.gz
Wed Feb  3 12:07:00 CST 2016 (I) Removing. Checksums are 2393381739 vs 2393381739
Wed Feb  3 12:07:00 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-1-20160107.gz vs c51-week-1-20160107.gz
Wed Feb  3 12:07:00 CST 2016 (I) Removing. Checksums are 259932587 vs 259932587
Wed Feb  3 12:07:00 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-2-20160114.gz vs charmonium-week-2-20160114.gz
Wed Feb  3 12:07:00 CST 2016 (I) Removing. Checksums are 218384547 vs 218384547
Wed Feb  3 12:07:00 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-2-20160114.gz vs c51-week-2-20160114.gz
Wed Feb  3 12:07:01 CST 2016 (I) Removing. Checksums are 2372418994 vs 2372418994
Wed Feb  3 12:07:01 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-2-20160114.gz vs disco-week-2-20160114.gz
Wed Feb  3 12:07:01 CST 2016 (I) Removing. Checksums are 4010689085 vs 4010689085
Wed Feb  3 12:07:01 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-2-20160114.gz vs fourpluseight-week-2-20160114.gz
Wed Feb  3 12:07:01 CST 2016 (I) Removing. Checksums are 1536640057 vs 1536640057
Wed Feb  3 12:07:01 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-2-20160114.gz vs mixbk-week-2-20160114.gz
Wed Feb  3 12:07:02 CST 2016 (I) Removing. Checksums are 3358277476 vs 3358277476
Wed Feb  3 12:07:02 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-2-20160114.gz vs nonperturbativestaggered-week-2-20160114.gz
Wed Feb  3 12:07:02 CST 2016 (I) Removing. Checksums are 2586593235 vs 2586593235
Wed Feb  3 12:07:02 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-2-20160114.gz vs ok-action-week-2-20160114.gz
Wed Feb  3 12:07:03 CST 2016 (I) Removing. Checksums are 2669561960 vs 2669561960
Wed Feb  3 12:07:03 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-2-20160114.gz vs rbcdwf-week-2-20160114.gz
Wed Feb  3 12:07:03 CST 2016 (I) Removing. Checksums are 115425591 vs 115425591
Wed Feb  3 12:07:03 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-2-20160114.gz vs hpqcd-week-2-20160114.gz
Wed Feb  3 12:07:03 CST 2016 (I) Removing. Checksums are 3935780579 vs 3935780579
Wed Feb  3 12:07:03 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-2-20160114.gz vs axial-week-2-20160114.gz
Wed Feb  3 12:07:04 CST 2016 (I) Removing. Checksums are 4279856720 vs 4279856720
Wed Feb  3 12:07:04 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-2-20160114.gz vs rhqbbar-week-2-20160114.gz
Wed Feb  3 12:07:04 CST 2016 (I) Removing. Checksums are 266787443 vs 266787443
Wed Feb  3 12:07:04 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-2-20160114.gz vs heavylight-week-2-20160114.gz
Wed Feb  3 12:07:04 CST 2016 (I) Removing. Checksums are 68087258 vs 68087258
Wed Feb  3 12:07:04 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-3-20160121.gz vs charmonium-week-3-20160121.gz
Wed Feb  3 12:07:05 CST 2016 (I) Removing. Checksums are 3201975380 vs 3201975380
Wed Feb  3 12:07:05 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-3-20160121.gz vs c51-week-3-20160121.gz
Wed Feb  3 12:07:05 CST 2016 (I) Removing. Checksums are 1260142674 vs 1260142674
Wed Feb  3 12:07:05 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-3-20160121.gz vs disco-week-3-20160121.gz
Wed Feb  3 12:07:05 CST 2016 (I) Removing. Checksums are 2831499310 vs 2831499310
Wed Feb  3 12:07:05 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-3-20160121.gz vs fourpluseight-week-3-20160121.gz
Wed Feb  3 12:07:06 CST 2016 (I) Removing. Checksums are 225002564 vs 225002564
Wed Feb  3 12:07:06 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-3-20160121.gz vs hpqcd-week-3-20160121.gz
Wed Feb  3 12:07:06 CST 2016 (I) Removing. Checksums are 2258386864 vs 2258386864
Wed Feb  3 12:07:06 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-3-20160121.gz vs mixbk-week-3-20160121.gz
Wed Feb  3 12:07:06 CST 2016 (I) Removing. Checksums are 2474000592 vs 2474000592
Wed Feb  3 12:07:06 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-3-20160121.gz vs ok-action-week-3-20160121.gz
Wed Feb  3 12:07:07 CST 2016 (I) Removing. Checksums are 154747309 vs 154747309
Wed Feb  3 12:07:07 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-3-20160121.gz vs nonperturbativestaggered-week-3-20160121.gz
Wed Feb  3 12:07:07 CST 2016 (I) Removing. Checksums are 388507985 vs 388507985
Wed Feb  3 12:07:07 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-3-20160121.gz vs rbcdwf-week-3-20160121.gz
Wed Feb  3 12:07:07 CST 2016 (I) Removing. Checksums are 382615602 vs 382615602
Wed Feb  3 12:07:07 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-3-20160121.gz vs axial-week-3-20160121.gz
Wed Feb  3 12:07:08 CST 2016 (I) Removing. Checksums are 3882272373 vs 3882272373
Wed Feb  3 12:07:08 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-3-20160121.gz vs rhqbbar-week-3-20160121.gz
Wed Feb  3 12:07:08 CST 2016 (I) Removing. Checksums are 1606419459 vs 1606419459
Wed Feb  3 12:07:08 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-3-20160121.gz vs heavylight-week-3-20160121.gz
Wed Feb  3 12:07:08 CST 2016 (I) Removing. Checksums are 787402680 vs 787402680
Wed Feb  3 12:07:08 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-4-20160128.gz vs c51-week-4-20160128.gz
Wed Feb  3 12:07:09 CST 2016 (I) Removing. Checksums are 1189101752 vs 1189101752
Wed Feb  3 12:07:09 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-4-20160128.gz vs charmonium-week-4-20160128.gz
Wed Feb  3 12:07:09 CST 2016 (I) Removing. Checksums are 2019846639 vs 2019846639
Wed Feb  3 12:07:09 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-4-20160128.gz vs disco-week-4-20160128.gz
Wed Feb  3 12:07:09 CST 2016 (I) Removing. Checksums are 2641314519 vs 2641314519
Wed Feb  3 12:07:09 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-4-20160128.gz vs fourpluseight-week-4-20160128.gz
Wed Feb  3 12:07:10 CST 2016 (I) Removing. Checksums are 149965703 vs 149965703
Wed Feb  3 12:07:10 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-4-20160128.gz vs hpqcd-week-4-20160128.gz
Wed Feb  3 12:07:10 CST 2016 (I) Removing. Checksums are 1946370591 vs 1946370591
Wed Feb  3 12:07:10 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-4-20160128.gz vs mixbk-week-4-20160128.gz
Wed Feb  3 12:07:10 CST 2016 (I) Removing. Checksums are 2044020069 vs 2044020069
Wed Feb  3 12:07:10 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-4-20160128.gz vs ok-action-week-4-20160128.gz
Wed Feb  3 12:07:11 CST 2016 (I) Removing. Checksums are 3095743034 vs 3095743034
Wed Feb  3 12:07:11 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-4-20160128.gz vs rbcdwf-week-4-20160128.gz
Wed Feb  3 12:07:11 CST 2016 (I) Removing. Checksums are 336871318 vs 336871318
Wed Feb  3 12:07:11 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-4-20160128.gz vs axial-week-4-20160128.gz
Wed Feb  3 12:07:11 CST 2016 (I) Removing. Checksums are 3756750631 vs 3756750631
Wed Feb  3 12:07:11 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-4-20160128.gz vs rhqbbar-week-4-20160128.gz
Wed Feb  3 12:07:12 CST 2016 (I) Removing. Checksums are 604327362 vs 604327362
Wed Feb  3 12:07:12 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-4-20160128.gz vs heavylight-week-4-20160128.gz
Wed Feb  3 12:07:12 CST 2016 (I) Removing. Checksums are 2042313975 vs 2042313975
Wed Feb  3 12:07:12 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-4-20160128.gz vs nonperturbativestaggered-week-4-20160128.gz
Wed Feb  3 12:07:13 CST 2016 (I) Removing. Checksums are 1802902312 vs 1802902312
Wed Feb  3 12:07:13 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-5-20160131.gz vs c51-week-5-20160131.gz
Wed Feb  3 12:07:13 CST 2016 (I) Removing. Checksums are 4139860148 vs 4139860148
Wed Feb  3 12:07:13 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-5-20160131.gz vs disco-week-5-20160131.gz
Wed Feb  3 12:07:13 CST 2016 (I) Removing. Checksums are 3311092616 vs 3311092616
Wed Feb  3 12:07:13 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-5-20160131.gz vs charmonium-week-5-20160131.gz
Wed Feb  3 12:07:14 CST 2016 (I) Removing. Checksums are 2309531583 vs 2309531583
Wed Feb  3 12:07:14 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-5-20160131.gz vs fourpluseight-week-5-20160131.gz
Wed Feb  3 12:07:14 CST 2016 (I) Removing. Checksums are 124929347 vs 124929347
Wed Feb  3 12:07:14 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-5-20160131.gz vs mixbk-week-5-20160131.gz
Wed Feb  3 12:07:14 CST 2016 (I) Removing. Checksums are 2890088644 vs 2890088644
Wed Feb  3 12:07:14 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-5-20160131.gz vs nonperturbativestaggered-week-5-20160131.gz
Wed Feb  3 12:07:15 CST 2016 (I) Removing. Checksums are 913591135 vs 913591135
Wed Feb  3 12:07:15 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-5-20160131.gz vs ok-action-week-5-20160131.gz
Wed Feb  3 12:07:15 CST 2016 (I) Removing. Checksums are 2775204969 vs 2775204969
Wed Feb  3 12:07:15 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-5-20160131.gz vs rbcdwf-week-5-20160131.gz
Wed Feb  3 12:07:15 CST 2016 (I) Removing. Checksums are 112671935 vs 112671935
Wed Feb  3 12:07:15 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-5-20160131.gz vs axial-week-5-20160131.gz
Wed Feb  3 12:07:16 CST 2016 (I) Removing. Checksums are 2065256347 vs 2065256347
Wed Feb  3 12:07:16 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-5-20160131.gz vs hpqcd-week-5-20160131.gz
Wed Feb  3 12:07:16 CST 2016 (I) Removing. Checksums are 695194048 vs 695194048
Wed Feb  3 12:07:16 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-5-20160131.gz vs heavylight-week-5-20160131.gz
Wed Feb  3 12:07:16 CST 2016 (I) Removing. Checksums are 2430992215 vs 2430992215
Wed Feb  3 12:07:16 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-5-20160131.gz vs rhqbbar-week-5-20160131.gz
Wed Feb  3 12:07:17 CST 2016 (I) Removing. Checksums are 346937683 vs 346937683
rm -f /nexsanpool2/filelist-201601.crc  /nexsanpool2/axial-day-1-20160101.gz /nexsanpool2/axial-week-1-20160107.gz /nexsanpool2/c51-day-1-20160101.gz /nexsanpool2/charmonium-day-1-20160101.gz /nexsanpool2/charmonium-week-1-20160107.gz /nexsanpool2/disco-day-1-20160101.gz /nexsanpool2/disco-week-1-20160107.gz /nexsanpool2/fourpluseight-day-1-20160101.gz /nexsanpool2/fourpluseight-week-1-20160107.gz /nexsanpool2/heavylight-day-1-20160101.gz /nexsanpool2/heavylight-week-1-20160107.gz /nexsanpool2/hpqcd-day-1-20160101.gz /nexsanpool2/hpqcd-week-1-20160107.gz /nexsanpool2/mixbk-day-1-20160101.gz /nexsanpool2/mixbk-week-1-20160107.gz /nexsanpool2/nonperturbativestaggered-day-1-20160101.gz /nexsanpool2/nonperturbativestaggered-week-1-20160107.gz /nexsanpool2/ok-action-day-1-20160101.gz /nexsanpool2/ok-action-week-1-20160107.gz /nexsanpool2/rbcdwf-day-1-20160101.gz /nexsanpool2/rbcdwf-week-1-20160107.gz /nexsanpool2/rhqbbar-day-1-20160101.gz /nexsanpool2/rhqbbar-week-1-20160107.gz /nexsanpool2/c51-week-1-20160107.gz /nexsanpool2/charmonium-week-2-20160114.gz /nexsanpool2/c51-week-2-20160114.gz /nexsanpool2/disco-week-2-20160114.gz /nexsanpool2/fourpluseight-week-2-20160114.gz /nexsanpool2/mixbk-week-2-20160114.gz /nexsanpool2/nonperturbativestaggered-week-2-20160114.gz /nexsanpool2/ok-action-week-2-20160114.gz /nexsanpool2/rbcdwf-week-2-20160114.gz /nexsanpool2/hpqcd-week-2-20160114.gz /nexsanpool2/axial-week-2-20160114.gz /nexsanpool2/rhqbbar-week-2-20160114.gz /nexsanpool2/heavylight-week-2-20160114.gz /nexsanpool2/charmonium-week-3-20160121.gz /nexsanpool2/c51-week-3-20160121.gz /nexsanpool2/disco-week-3-20160121.gz /nexsanpool2/fourpluseight-week-3-20160121.gz /nexsanpool2/hpqcd-week-3-20160121.gz /nexsanpool2/mixbk-week-3-20160121.gz /nexsanpool2/ok-action-week-3-20160121.gz /nexsanpool2/nonperturbativestaggered-week-3-20160121.gz /nexsanpool2/rbcdwf-week-3-20160121.gz /nexsanpool2/axial-week-3-20160121.gz /nexsanpool2/rhqbbar-week-3-20160121.gz /nexsanpool2/heavylight-week-3-20160121.gz /nexsanpool2/c51-week-4-20160128.gz /nexsanpool2/charmonium-week-4-20160128.gz /nexsanpool2/disco-week-4-20160128.gz /nexsanpool2/fourpluseight-week-4-20160128.gz /nexsanpool2/hpqcd-week-4-20160128.gz /nexsanpool2/mixbk-week-4-20160128.gz /nexsanpool2/ok-action-week-4-20160128.gz /nexsanpool2/rbcdwf-week-4-20160128.gz /nexsanpool2/axial-week-4-20160128.gz /nexsanpool2/rhqbbar-week-4-20160128.gz /nexsanpool2/heavylight-week-4-20160128.gz /nexsanpool2/nonperturbativestaggered-week-4-20160128.gz /nexsanpool2/c51-week-5-20160131.gz /nexsanpool2/disco-week-5-20160131.gz /nexsanpool2/charmonium-week-5-20160131.gz /nexsanpool2/fourpluseight-week-5-20160131.gz /nexsanpool2/mixbk-week-5-20160131.gz /nexsanpool2/nonperturbativestaggered-week-5-20160131.gz /nexsanpool2/ok-action-week-5-20160131.gz /nexsanpool2/rbcdwf-week-5-20160131.gz /nexsanpool2/axial-week-5-20160131.gz /nexsanpool2/hpqcd-week-5-20160131.gz /nexsanpool2/heavylight-week-5-20160131.gz /nexsanpool2/rhqbbar-week-5-20160131.gz
[root@dslustre21 ~]# 

I had to do some fixing to the script but now it's commited in GIT and tomorrow the files should be gone, which is very good because space is tight!

[root@dslustre21 ~]# zpool list
NAME             SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
nexsanpool1     16.2T  7.74T  8.51T    47%  1.00x  ONLINE  -
nexsanpool2     16.2T  15.1T  1.14T    92%  1.00x  ONLINE  -
projectzfspool  7.25T  5.56T  1.69T    76%  1.00x  ONLINE  -
[root@dslustre21 ~]# 

#15 Updated by Gerard Bernabeu Altayo over 3 years ago

The only pending thing now is to watch this for 1 month, if it all works then we're good to go!

#16 Updated by Gerard Bernabeu Altayo over 3 years ago

  • Status changed from New to Resolved

it worked just fine for the February backups, closing ticket.



Also available in: Atom PDF