Bug #11372
HPC: try to optimize the ZFS cleanup process
Start date:
01/08/2016
Due date:
% Done:
90%
Estimated time:
Description
At PIC I've seen how they're managing the ZFS snapshots and seems like removing incremental snapshots shouldn't be an issue. I should make some tests as it would simplify a few processes.
History
#1 Updated by Gerard Bernabeu Altayo almost 5 years ago
I will experience with nexsandata1 as it's easy to reproduce:
[root@dslustre21 ~]# zfs diff nexsanpool1/rhqbbar@day-3-20160103 nexsanpool1/rhqbbar@day-4-20160104 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500 M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/DWF_Prop_exact_4545_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/QLUA_exact_4545_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/DWF_Prop_exact_4565_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/QLUA_exact_4565_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/DWF_Prop_exact_4570_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/QLUA_exact_4570_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/DWF_Prop_exact_4575_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/QLUA_exact_4575_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/DWF_Prop_exact_4590_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/QLUA_exact_4590_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/DWF_Prop_exact_4610_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/QLUA_exact_4610_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/DWF_Prop_exact_4635_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/QLUA_exact_4635_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/DWF_Prop_exact_4650_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/QLUA_exact_4650_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/DWF_Prop_exact_4665_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/QLUA_exact_4665_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/DWF_Prop_exact_4685_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/QLUA_exact_4685_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/DWF_Prop_exact_4700_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/QLUA_exact_4700_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/DWF_Prop_exact_4715_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/QLUA_exact_4715_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/DWF_Prop_exact_4720_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/QLUA_exact_4720_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/DWF_Prop_exact_4725_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/QLUA_exact_4725_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/DWF_Prop_exact_4730_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/QLUA_exact_4730_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/DWF_Prop_exact_4740_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/QLUA_exact_4740_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/DWF_Prop_exact_4750_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/QLUA_exact_4750_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/DWF_Prop_exact_4760_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/QLUA_exact_4760_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/DWF_Prop_exact_4765_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/QLUA_exact_4765_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/DWF_Prop_exact_4770_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/QLUA_exact_4770_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/DWF_Prop_exact_4775_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/QLUA_exact_4775_m0.030_x0_y0_z0_t0.job.sh M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/DWF_Prop_exact_7275_m0.030_x0_y0_z0_t0.qlua M /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/QLUA_exact_7275_m0.030_x0_y0_z0_t0.job.sh + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/Q24c_4665_m0.030_x0_y0_z0_t0-1351397.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/Q24c_4720_m0.030_x0_y0_z0_t0-1351401.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/Q24c_4725_m0.030_x0_y0_z0_t0-1351402.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/Q24c_4750_m0.030_x0_y0_z0_t0-1351405.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/Q24c_4760_m0.030_x0_y0_z0_t0-1351406.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/Q24c_4770_m0.030_x0_y0_z0_t0-1351408.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/Q24c_4775_m0.030_x0_y0_z0_t0-1351409.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/DWF_Prop_exact_6500_m0.030_x0_y0_z0_t0.qlua + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/QLUA_exact_6500_m0.030_x0_y0_z0_t0.job.sh + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/Q24c_6500_m0.030_x0_y0_z0_t0-1351410.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/Q24c_4570_m0.030_x0_y0_z0_t0-1351391.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/Q24c_4545_m0.030_x0_y0_z0_t0-1351389.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/Q24c_4565_m0.030_x0_y0_z0_t0-1351390.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/Q24c_4575_m0.030_x0_y0_z0_t0-1351392.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/Q24c_4715_m0.030_x0_y0_z0_t0-1351400.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/Q24c_4590_m0.030_x0_y0_z0_t0-1351393.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/Q24c_4610_m0.030_x0_y0_z0_t0-1351394.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/Q24c_4635_m0.030_x0_y0_z0_t0-1351395.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/Q24c_4685_m0.030_x0_y0_z0_t0-1351398.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/Q24c_4700_m0.030_x0_y0_z0_t0-1351399.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/Q24c_4730_m0.030_x0_y0_z0_t0-1351403.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/Q24c_4740_m0.030_x0_y0_z0_t0-1351404.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/Q24c_4765_m0.030_x0_y0_z0_t0-1351407.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/Q24c_4650_m0.030_x0_y0_z0_t0-1351396.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0-1351411.ds2.fnal.gov.out + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4750/Q24c_4750_m0.030_x0_y0_z0_t0.o1351405 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4665/Q24c_4665_m0.030_x0_y0_z0_t0.o1351397 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4725/Q24c_4725_m0.030_x0_y0_z0_t0.o1351402 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4775/Q24c_4775_m0.030_x0_y0_z0_t0.o1351409 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_6500/Q24c_6500_m0.030_x0_y0_z0_t0.o1351410 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4720/Q24c_4720_m0.030_x0_y0_z0_t0.o1351401 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4770/Q24c_4770_m0.030_x0_y0_z0_t0.o1351408 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4545/Q24c_4545_m0.030_x0_y0_z0_t0.o1351389 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4760/Q24c_4760_m0.030_x0_y0_z0_t0.o1351406 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4610/Q24c_4610_m0.030_x0_y0_z0_t0.o1351394 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4565/Q24c_4565_m0.030_x0_y0_z0_t0.o1351390 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4715/Q24c_4715_m0.030_x0_y0_z0_t0.o1351400 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4575/Q24c_4575_m0.030_x0_y0_z0_t0.o1351392 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4730/Q24c_4730_m0.030_x0_y0_z0_t0.o1351403 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4685/Q24c_4685_m0.030_x0_y0_z0_t0.o1351398 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4590/Q24c_4590_m0.030_x0_y0_z0_t0.o1351393 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4700/Q24c_4700_m0.030_x0_y0_z0_t0.o1351399 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4635/Q24c_4635_m0.030_x0_y0_z0_t0.o1351395 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4570/Q24c_4570_m0.030_x0_y0_z0_t0.o1351391 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4740/Q24c_4740_m0.030_x0_y0_z0_t0.o1351404 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4765/Q24c_4765_m0.030_x0_y0_z0_t0.o1351407 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_4650/Q24c_4650_m0.030_x0_y0_z0_t0.o1351396 + /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411 [root@dslustre21 ~]# [root@dslustre21 ~]# ll /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411 -rw-r--r-- 1 witzel rhqbbar 13 Jan 4 04:42 /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411 [root@dslustre21 ~]# cat /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411 PBS epilogue [root@dslustre21 ~]#
Now I will destroy the dataset where this file was created: nexsanpool1/rhqbbar@day-4-20160104
[root@dslustre21 ~]# zfs list -t snapshot | grep rhqbbar nexsanpool1/rhqbbar@day-1-20160101 63.2M - 731G - nexsanpool1/rhqbbar@day-2-20160102 15.0M - 731G - nexsanpool1/rhqbbar@day-3-20160103 432K - 731G - nexsanpool1/rhqbbar@day-4-20160104 208K - 731G - nexsanpool1/rhqbbar@day-5-20160105 1007K - 712G - nexsanpool1/rhqbbar@day-6-20160106 791K - 713G - nexsanpool1/rhqbbar@day-7-20160107 0 - 713G - nexsanpool2/rhqbbar@day-1-20160101 26.8G - 731G - nexsanpool2/rhqbbar@week-1-20160107 0 - 713G - projectzfspool/projectzfs/rhqbbar@day-1-20160101 31.6M - 818G - projectzfspool/projectzfs/rhqbbar@day-2-20160102 7.52M - 818G - projectzfspool/projectzfs/rhqbbar@day-3-20160103 216K - 818G - projectzfspool/projectzfs/rhqbbar@day-4-20160104 104K - 818G - projectzfspool/projectzfs/rhqbbar@day-5-20160105 512K - 801G - projectzfspool/projectzfs/rhqbbar@day-6-20160106 412K - 801G - projectzfspool/projectzfs/rhqbbar@day-7-20160107 1.01M - 801G - [root@dslustre21 ~]# zfs destroy nexsanpool1/rhqbbar@day-4-20160104 [root@dslustre21 ~]# echo $? 0 [root@dslustre21 ~]# zfs list -t snapshot | grep rhqbbar nexsanpool1/rhqbbar@day-1-20160101 63.2M - 731G - nexsanpool1/rhqbbar@day-2-20160102 15.0M - 731G - nexsanpool1/rhqbbar@day-3-20160103 432K - 731G - nexsanpool1/rhqbbar@day-5-20160105 1007K - 712G - nexsanpool1/rhqbbar@day-6-20160106 791K - 713G - nexsanpool1/rhqbbar@day-7-20160107 176K - 713G - nexsanpool2/rhqbbar@day-1-20160101 26.8G - 731G - nexsanpool2/rhqbbar@week-1-20160107 0 - 713G - projectzfspool/projectzfs/rhqbbar@day-1-20160101 31.6M - 818G - projectzfspool/projectzfs/rhqbbar@day-2-20160102 7.52M - 818G - projectzfspool/projectzfs/rhqbbar@day-3-20160103 216K - 818G - projectzfspool/projectzfs/rhqbbar@day-4-20160104 104K - 818G - projectzfspool/projectzfs/rhqbbar@day-5-20160105 512K - 801G - projectzfspool/projectzfs/rhqbbar@day-6-20160106 412K - 801G - projectzfspool/projectzfs/rhqbbar@day-7-20160107 1.01M - 801G - [root@dslustre21 ~]#
The file still exists, this shows that there is no data loss:
[root@dslustre21 ~]# ll /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411 -rw-r--r-- 1 witzel rhqbbar 13 Jan 4 04:42 /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411 [root@dslustre21 ~]# cat /nexsanpool1/rhqbbar/24c/l24t64f21b213m005m040/Production_m0.005/config_7275/Q24c_7275_m0.030_x0_y0_z0_t0.o1351411 PBS epilogue [root@dslustre21 ~]#
Of course now I'm not capable of restoring the snapshot by applying it because the changes are already in:
[root@dslustre21 ~]# /sbin/zfs send -v -i projectzfspool/projectzfs/rhqbbar@day-3-20160103 projectzfspool/projectzfs/rhqbbar@day-4-20160104 | /sbin/zfs recv nexsanpool1/rhqbbar@day-4-20160104 send from @day-3-20160103 to projectzfspool/projectzfs/rhqbbar@day-4-20160104 estimated size is 1.59M total estimated size is 1.59M TIME SENT SNAPSHOT cannot receive incremental stream: destination nexsanpool1/rhqbbar has been modified since most recent snapshot warning: cannot send 'projectzfspool/projectzfs/rhqbbar@day-4-20160104': Broken pipe
And forcing it rolls it back:
[root@dslustre21 ~]# /sbin/zfs send -v -i projectzfspool/projectzfs/rhqbbar@day-3-20160103 projectzfspool/projectzfs/rhqbbar@day-4-20160104 | /sbin/zfs recv -F nexsanpool1/rhqbbar@day-4-20160104 send from @day-3-20160103 to projectzfspool/projectzfs/rhqbbar@day-4-20160104 estimated size is 1.59M total estimated size is 1.59M TIME SENT SNAPSHOT [root@dslustre21 ~]# zfs list -t snapshot | grep rhqbbar nexsanpool1/rhqbbar@day-1-20160101 63.2M - 731G - nexsanpool1/rhqbbar@day-2-20160102 15.0M - 731G - nexsanpool1/rhqbbar@day-3-20160103 432K - 731G - nexsanpool1/rhqbbar@day-4-20160104 0 - 731G - nexsanpool2/rhqbbar@day-1-20160101 26.8G - 731G - nexsanpool2/rhqbbar@week-1-20160107 0 - 713G - projectzfspool/projectzfs/rhqbbar@day-1-20160101 31.6M - 818G - projectzfspool/projectzfs/rhqbbar@day-2-20160102 7.52M - 818G - projectzfspool/projectzfs/rhqbbar@day-3-20160103 216K - 818G - projectzfspool/projectzfs/rhqbbar@day-4-20160104 104K - 818G - projectzfspool/projectzfs/rhqbbar@day-5-20160105 512K - 801G - projectzfspool/projectzfs/rhqbbar@day-6-20160106 412K - 801G - projectzfspool/projectzfs/rhqbbar@day-7-20160107 1.01M - 801G - [root@dslustre21 ~]#
Now I will just redo the backups by running the script again:
[root@dslustre21 ~]# echo rhqbbar | /root/zfs-scripts/replica-zfs-snapshot.pl Jan 08 10:59:44 (I) Reading dataset list from stdin .. Jan 08 10:59:44 (I) Making daily snapshots - Snapshot date tag is 8 Jan 08 10:59:44 (I) Replicating snapshots for rhqbbar Jan 08 10:59:44 (I) Creating daily snapshot: /sbin/zfs send -v -i projectzfspool/projectzfs/rhqbbar@day-4-20160104 projectzfspool/projectzfs/rhqbbar@day-5-20160105 | /sbin/zfs recv -F nexsanpool1/rhqbbar@day-5-20160105 send from @day-4-20160104 to projectzfspool/projectzfs/rhqbbar@day-5-20160105 estimated size is 7.63G total estimated size is 7.63G TIME SENT SNAPSHOT 10:59:46 57.0K projectzfspool/projectzfs/rhqbbar@day-5-20160105 10:59:47 26.6M projectzfspool/projectzfs/rhqbbar@day-5-20160105 10:59:48 102M projectzfspool/projectzfs/rhqbbar@day-5-20160105 10:59:49 206M projectzfspool/projectzfs/rhqbbar@day-5-20160105 10:59:50 306M projectzfspool/projectzfs/rhqbbar@day-5-20160105 ...
#2 Updated by Gerard Bernabeu Altayo almost 5 years ago
Conclusion is that the zpool destroy -R is not needed and I can only destroy the snapshots and let the new snapshots come in.
#3 Updated by Gerard Bernabeu Altayo almost 5 years ago
Getting the files to remove for this month:
[root@dslustre21 zfs-scripts]# bash ./delete-gzips.sh > FilesToRemove.20160111.log 2>&1 [root@dslustre21 zfs-scripts]# screen [detached]
I need to implement the efficient way of doing this (calculate CRC after file generation so that it's in cache!)
#4 Updated by Gerard Bernabeu Altayo almost 5 years ago
The script failed due to SSH permission issues from within the screen:
Permission denied (gssapi-keyex,gssapi-with-mic,keyboard-interactive).
Since I have all the local checksums calculated I will create the final script that processes and removes the files :)
#5 Updated by Gerard Bernabeu Altayo almost 5 years ago
I've created the list:
[root@dslustre21 zfs-scripts]# head /nexsanpool2/filelist-201512.crc /nexsanpool2/axial-day-1-20160101.gz 364767830 /nexsanpool2/axial-week-1-20160107.gz 3921114051 /nexsanpool2/axial-week-3-20151221.gz 2417836686 /nexsanpool2/axial-week-4-20151228.gz 2621114247 /nexsanpool2/axial-week-5-20151231.gz 1494400073 /nexsanpool2/c51-day-1-20151201.gz 2265708214 /nexsanpool2/c51-day-1-20160101.gz 2900074934 /nexsanpool2/c51-week-1-20151207.gz 3635232688 /nexsanpool2/c51-week-1-20160107.gz 1980908136 /nexsanpool2/c51-week-2-20151214.gz 1045316058
Will now adapt the script to read this file, and then I have to generate them :).
#6 Updated by Gerard Bernabeu Altayo almost 5 years ago
The script to remove the files is ready, I am not uncomenting the final 'rm' because next month I want to launch the rm after careful consideration, but if it works this could be in a cron that runs the 2nd week of the month or so...
[root@dslustre21 zfs-scripts]# bash ./delete-gzips.sh
There are 63 /nexsanpool2/filelist-201512.crc files to be considered for removal
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-week-3-20151221.gz vs axial-week-3-20151221.gz
INFO: Removing. Checksums are 2417836686 vs 2417836686
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-week-4-20151228.gz vs axial-week-4-20151228.gz
INFO: Removing. Checksums are 2621114247 vs 2621114247
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-week-5-20151231.gz vs axial-week-5-20151231.gz
INFO: Removing. Checksums are 1494400073 vs 1494400073
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-day-1-20151201.gz vs c51-day-1-20151201.gz
INFO: Removing. Checksums are 2265708214 vs 2265708214
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-1-20151207.gz vs c51-week-1-20151207.gz
INFO: Removing. Checksums are 3635232688 vs 3635232688
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-2-20151214.gz vs c51-week-2-20151214.gz
INFO: Removing. Checksums are 1045316058 vs 1045316058
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-3-20151221.gz vs c51-week-3-20151221.gz
INFO: Removing. Checksums are 1954234560 vs 1954234560
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-4-20151228.gz vs c51-week-4-20151228.gz
INFO: Removing. Checksums are 1486962691 vs 1486962691
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/c51-week-5-20151231.gz vs c51-week-5-20151231.gz
INFO: Removing. Checksums are 1609711399 vs 1609711399
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-day-1-20151201.gz vs charmonium-day-1-20151201.gz
INFO: Removing. Checksums are 1893491997 vs 1893491997
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-1-20151207.gz vs charmonium-week-1-20151207.gz
INFO: Removing. Checksums are 1113410888 vs 1113410888
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-2-20151214.gz vs charmonium-week-2-20151214.gz
INFO: Removing. Checksums are 1044791556 vs 1044791556
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-3-20151221.gz vs charmonium-week-3-20151221.gz
INFO: Removing. Checksums are 1059013925 vs 1059013925
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-4-20151228.gz vs charmonium-week-4-20151228.gz
INFO: Removing. Checksums are 591282615 vs 591282615
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/charmonium-week-5-20151231.gz vs charmonium-week-5-20151231.gz
INFO: Removing. Checksums are 3002811213 vs 3002811213
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-day-1-20151201.gz vs disco-day-1-20151201.gz
INFO: Removing. Checksums are 1917477495 vs 1917477495
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-1-20151207.gz vs disco-week-1-20151207.gz
INFO: Removing. Checksums are 98124554 vs 98124554
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-2-20151214.gz vs disco-week-2-20151214.gz
INFO: Removing. Checksums are 2869313602 vs 2869313602
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-3-20151221.gz vs disco-week-3-20151221.gz
INFO: Removing. Checksums are 688406556 vs 688406556
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-4-20151228.gz vs disco-week-4-20151228.gz
INFO: Removing. Checksums are 3814671458 vs 3814671458
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/disco-week-5-20151231.gz vs disco-week-5-20151231.gz
INFO: Removing. Checksums are 3636611166 vs 3636611166
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-day-1-20151201.gz vs fourpluseight-day-1-20151201.gz
INFO: Removing. Checksums are 3271725874 vs 3271725874
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-1-20151207.gz vs fourpluseight-week-1-20151207.gz
INFO: Removing. Checksums are 959989706 vs 959989706
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-2-20151214.gz vs fourpluseight-week-2-20151214.gz
INFO: Removing. Checksums are 3852224621 vs 3852224621
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-3-20151221.gz vs fourpluseight-week-3-20151221.gz
INFO: Removing. Checksums are 1408695196 vs 1408695196
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-4-20151228.gz vs fourpluseight-week-4-20151228.gz
INFO: Removing. Checksums are 118311785 vs 118311785
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/fourpluseight-week-5-20151231.gz vs fourpluseight-week-5-20151231.gz
INFO: Removing. Checksums are 1839548120 vs 1839548120
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-day-1-20151201.gz vs hpqcd-day-1-20151201.gz
INFO: Removing. Checksums are 498550980 vs 498550980
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-1-20151207.gz vs hpqcd-week-1-20151207.gz
INFO: Removing. Checksums are 2402566473 vs 2402566473
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-2-20151214.gz vs hpqcd-week-2-20151214.gz
INFO: Removing. Checksums are 1893744883 vs 1893744883
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-3-20151221.gz vs hpqcd-week-3-20151221.gz
INFO: Removing. Checksums are 383154737 vs 383154737
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-4-20151228.gz vs hpqcd-week-4-20151228.gz
INFO: Removing. Checksums are 4135273090 vs 4135273090
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/hpqcd-week-5-20151231.gz vs hpqcd-week-5-20151231.gz
INFO: Removing. Checksums are 4223353350 vs 4223353350
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-day-1-20151201.gz vs mixbk-day-1-20151201.gz
INFO: Removing. Checksums are 495425356 vs 495425356
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-1-20151207.gz vs mixbk-week-1-20151207.gz
INFO: Removing. Checksums are 4274602064 vs 4274602064
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-2-20151214.gz vs mixbk-week-2-20151214.gz
INFO: Removing. Checksums are 1894860552 vs 1894860552
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-3-20151221.gz vs mixbk-week-3-20151221.gz
INFO: Removing. Checksums are 3871489434 vs 3871489434
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-4-20151228.gz vs mixbk-week-4-20151228.gz
INFO: Removing. Checksums are 1193362265 vs 1193362265
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/mixbk-week-5-20151231.gz vs mixbk-week-5-20151231.gz
INFO: Removing. Checksums are 1567245158 vs 1567245158
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-day-1-20151201.gz vs nonperturbativestaggered-day-1-20151201.gz
INFO: Removing. Checksums are 1414143160 vs 1414143160
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-1-20151207.gz vs nonperturbativestaggered-week-1-20151207.gz
INFO: Removing. Checksums are 1555842468 vs 1555842468
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-2-20151214.gz vs nonperturbativestaggered-week-2-20151214.gz
INFO: Removing. Checksums are 2831435499 vs 2831435499
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-3-20151221.gz vs nonperturbativestaggered-week-3-20151221.gz
INFO: Removing. Checksums are 3492496316 vs 3492496316
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-4-20151228.gz vs nonperturbativestaggered-week-4-20151228.gz
INFO: Removing. Checksums are 3638249525 vs 3638249525
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/nonperturbativestaggered-week-5-20151231.gz vs nonperturbativestaggered-week-5-20151231.gz
INFO: Removing. Checksums are 1580354178 vs 1580354178
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-day-1-20151201.gz vs ok-action-day-1-20151201.gz
INFO: Removing. Checksums are 3381800064 vs 3381800064
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-1-20151207.gz vs ok-action-week-1-20151207.gz
INFO: Removing. Checksums are 3284813758 vs 3284813758
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-2-20151214.gz vs ok-action-week-2-20151214.gz
INFO: Removing. Checksums are 2853716679 vs 2853716679
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-3-20151221.gz vs ok-action-week-3-20151221.gz
INFO: Removing. Checksums are 2734047468 vs 2734047468
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-4-20151228.gz vs ok-action-week-4-20151228.gz
INFO: Removing. Checksums are 3036693111 vs 3036693111
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ok-action-week-5-20151231.gz vs ok-action-week-5-20151231.gz
INFO: Removing. Checksums are 2246263745 vs 2246263745
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-day-1-20151201.gz vs rbcdwf-day-1-20151201.gz
INFO: Removing. Checksums are 3612825378 vs 3612825378
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-1-20151207.gz vs rbcdwf-week-1-20151207.gz
INFO: Removing. Checksums are 322453349 vs 322453349
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-2-20151214.gz vs rbcdwf-week-2-20151214.gz
INFO: Removing. Checksums are 2662220017 vs 2662220017
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-3-20151221.gz vs rbcdwf-week-3-20151221.gz
INFO: Removing. Checksums are 2247966270 vs 2247966270
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-4-20151228.gz vs rbcdwf-week-4-20151228.gz
INFO: Removing. Checksums are 21250907 vs 21250907
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rbcdwf-week-5-20151231.gz vs rbcdwf-week-5-20151231.gz
INFO: Removing. Checksums are 1982546892 vs 1982546892
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-day-1-20151201.gz vs rhqbbar-day-1-20151201.gz
INFO: Removing. Checksums are 91887815 vs 91887815
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-1-20151207.gz vs rhqbbar-week-1-20151207.gz
INFO: Removing. Checksums are 3725737770 vs 3725737770
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-2-20151214.gz vs rhqbbar-week-2-20151214.gz
INFO: Removing. Checksums are 1445346914 vs 1445346914
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-3-20151221.gz vs rhqbbar-week-3-20151221.gz
INFO: Removing. Checksums are 3672809767 vs 3672809767
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-4-20151228.gz vs rhqbbar-week-4-20151228.gz
INFO: Removing. Checksums are 1304247893 vs 1304247893
Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/rhqbbar-week-5-20151231.gz vs rhqbbar-week-5-20151231.gz
INFO: Removing. Checksums are 2179975191 vs 2179975191
rm -f /nexsanpool2/filelist-201512.crc /nexsanpool2/axial-week-3-20151221.gz /nexsanpool2/axial-week-4-20151228.gz /nexsanpool2/axial-week-5-20151231.gz /nexsanpool2/c51-day-1-20151201.gz /nexsanpool2/c51-week-1-20151207.gz /nexsanpool2/c51-week-2-20151214.gz /nexsanpool2/c51-week-3-20151221.gz /nexsanpool2/c51-week-4-20151228.gz /nexsanpool2/c51-week-5-20151231.gz /nexsanpool2/charmonium-day-1-20151201.gz /nexsanpool2/charmonium-week-1-20151207.gz /nexsanpool2/charmonium-week-2-20151214.gz /nexsanpool2/charmonium-week-3-20151221.gz /nexsanpool2/charmonium-week-4-20151228.gz /nexsanpool2/charmonium-week-5-20151231.gz /nexsanpool2/disco-day-1-20151201.gz /nexsanpool2/disco-week-1-20151207.gz /nexsanpool2/disco-week-2-20151214.gz /nexsanpool2/disco-week-3-20151221.gz /nexsanpool2/disco-week-4-20151228.gz /nexsanpool2/disco-week-5-20151231.gz /nexsanpool2/fourpluseight-day-1-20151201.gz /nexsanpool2/fourpluseight-week-1-20151207.gz /nexsanpool2/fourpluseight-week-2-20151214.gz /nexsanpool2/fourpluseight-week-3-20151221.gz /nexsanpool2/fourpluseight-week-4-20151228.gz /nexsanpool2/fourpluseight-week-5-20151231.gz /nexsanpool2/hpqcd-day-1-20151201.gz /nexsanpool2/hpqcd-week-1-20151207.gz /nexsanpool2/hpqcd-week-2-20151214.gz /nexsanpool2/hpqcd-week-3-20151221.gz /nexsanpool2/hpqcd-week-4-20151228.gz /nexsanpool2/hpqcd-week-5-20151231.gz /nexsanpool2/mixbk-day-1-20151201.gz /nexsanpool2/mixbk-week-1-20151207.gz /nexsanpool2/mixbk-week-2-20151214.gz /nexsanpool2/mixbk-week-3-20151221.gz /nexsanpool2/mixbk-week-4-20151228.gz /nexsanpool2/mixbk-week-5-20151231.gz /nexsanpool2/nonperturbativestaggered-day-1-20151201.gz /nexsanpool2/nonperturbativestaggered-week-1-20151207.gz /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz /nexsanpool2/nonperturbativestaggered-week-4-20151228.gz /nexsanpool2/nonperturbativestaggered-week-5-20151231.gz /nexsanpool2/ok-action-day-1-20151201.gz /nexsanpool2/ok-action-week-1-20151207.gz /nexsanpool2/ok-action-week-2-20151214.gz /nexsanpool2/ok-action-week-3-20151221.gz /nexsanpool2/ok-action-week-4-20151228.gz /nexsanpool2/ok-action-week-5-20151231.gz /nexsanpool2/rbcdwf-day-1-20151201.gz /nexsanpool2/rbcdwf-week-1-20151207.gz /nexsanpool2/rbcdwf-week-2-20151214.gz /nexsanpool2/rbcdwf-week-3-20151221.gz /nexsanpool2/rbcdwf-week-4-20151228.gz /nexsanpool2/rbcdwf-week-5-20151231.gz /nexsanpool2/rhqbbar-day-1-20151201.gz /nexsanpool2/rhqbbar-week-1-20151207.gz /nexsanpool2/rhqbbar-week-2-20151214.gz /nexsanpool2/rhqbbar-week-3-20151221.gz /nexsanpool2/rhqbbar-week-4-20151228.gz /nexsanpool2/rhqbbar-week-5-20151231.gz
[root@dslustre21 zfs-scripts]# rm -f /nexsanpool2/filelist-201512.crc /nexsanpool2/axial-week-3-20151221.gz /nexsanpool2/axial-week-4-20151228.gz /nexsanpool2/axial-week-5-20151231.gz /nexsanpool2/c51-day-1-20151201.gz /nexsanpool2/c51-week-1-20151207.gz /nexsanpool2/c51-week-2-20151214.gz /nexsanpool2/c51-week-3-20151221.gz /nexsanpool2/c51-week-4-20151228.gz /nexsanpool2/c51-week-5-20151231.gz /nexsanpool2/charmonium-day-1-20151201.gz /nexsanpool2/charmonium-week-1-20151207.gz /nexsanpool2/charmonium-week-2-20151214.gz /nexsanpool2/charmonium-week-3-20151221.gz /nexsanpool2/charmonium-week-4-20151228.gz /nexsanpool2/charmonium-week-5-20151231.gz /nexsanpool2/disco-day-1-20151201.gz /nexsanpool2/disco-week-1-20151207.gz /nexsanpool2/disco-week-2-20151214.gz /nexsanpool2/disco-week-3-20151221.gz /nexsanpool2/disco-week-4-20151228.gz /nexsanpool2/disco-week-5-20151231.gz /nexsanpool2/fourpluseight-day-1-20151201.gz /nexsanpool2/fourpluseight-week-1-20151207.gz /nexsanpool2/fourpluseight-week-2-20151214.gz /nexsanpool2/fourpluseight-week-3-20151221.gz /nexsanpool2/fourpluseight-week-4-20151228.gz /nexsanpool2/fourpluseight-week-5-20151231.gz /nexsanpool2/hpqcd-day-1-20151201.gz /nexsanpool2/hpqcd-week-1-20151207.gz /nexsanpool2/hpqcd-week-2-20151214.gz /nexsanpool2/hpqcd-week-3-20151221.gz /nexsanpool2/hpqcd-week-4-20151228.gz /nexsanpool2/hpqcd-week-5-20151231.gz /nexsanpool2/mixbk-day-1-20151201.gz /nexsanpool2/mixbk-week-1-20151207.gz /nexsanpool2/mixbk-week-2-20151214.gz /nexsanpool2/mixbk-week-3-20151221.gz /nexsanpool2/mixbk-week-4-20151228.gz /nexsanpool2/mixbk-week-5-20151231.gz /nexsanpool2/nonperturbativestaggered-day-1-20151201.gz /nexsanpool2/nonperturbativestaggered-week-1-20151207.gz /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz /nexsanpool2/nonperturbativestaggered-week-4-20151228.gz /nexsanpool2/nonperturbativestaggered-week-5-20151231.gz /nexsanpool2/ok-action-day-1-20151201.gz /nexsanpool2/ok-action-week-1-20151207.gz /nexsanpool2/ok-action-week-2-20151214.gz /nexsanpool2/ok-action-week-3-20151221.gz /nexsanpool2/ok-action-week-4-20151228.gz /nexsanpool2/ok-action-week-5-20151231.gz /nexsanpool2/rbcdwf-day-1-20151201.gz /nexsanpool2/rbcdwf-week-1-20151207.gz /nexsanpool2/rbcdwf-week-2-20151214.gz /nexsanpool2/rbcdwf-week-3-20151221.gz /nexsanpool2/rbcdwf-week-4-20151228.gz /nexsanpool2/rbcdwf-week-5-20151231.gz /nexsanpool2/rhqbbar-day-1-20151201.gz /nexsanpool2/rhqbbar-week-1-20151207.gz /nexsanpool2/rhqbbar-week-2-20151214.gz /nexsanpool2/rhqbbar-week-3-20151221.gz /nexsanpool2/rhqbbar-week-4-20151228.gz /nexsanpool2/rhqbbar-week-5-20151231.gz
#7 Updated by Gerard Bernabeu Altayo almost 5 years ago
ecrc creation automated with the gzip creation:
[root@dslustre21 nexsanpool2]# echo c51 | /root/zfs-scripts/replica-zfs-snapshot.pl Jan 12 17:37:33 (I) Reading dataset list from stdin .. Jan 12 17:37:33 (I) Making daily snapshots - Snapshot date tag is 12 Jan 12 17:37:33 (I) Replicating snapshots for c51 Jan 12 17:37:33 (I) Making weekly snapshots Jan 12 17:37:33 (I) Checking weekly snapshots for c51 Jan 12 17:37:33 (I) There should exist week-1-20160107 . Need to create Jan 12 17:37:33 (I) Making GZIPs Jan 12 17:37:33 (I) Making GZIP for c51 Jan 12 17:37:33 (I) /nexsanpool2/c51-day-1-20160101.gz already exists Jan 12 17:37:33 (I) Creating /nexsanpool2/c51-week-1-20160107.gz with: (/sbin/zfs send -i nexsanpool2/c51@day-1-20160101 nexsanpool2/c51@week-1-20160107 | gzip > /nexsanpool2/c51-week-1-20160107.gz && (echo -n 'c51-week-1-20160107.gz ' && ecrc /nexsanpool2/c51-week-1-20160107.gz | cut -d' ' -f 2 ) >> /nexsanpool2/filelist-201601.crc )&
Now what's missing is to fully automate the copies to tape and gzip removal.
#8 Updated by Gerard Bernabeu Altayo almost 5 years ago
In order to run the log rotation script I should use a line like the following (from the monthly cron):
/root/zfs-scripts/delete-gzips.sh >> /root/zfs-scripts/delete-zfs-gzips.log 2>&1
Adding monitoring for this error logs:
[root@dslustre21 check-mk-agent]# diff logwatch.cfg ../logwatch.cfg 58a59,64 > > #Checks for ZFS backups > /root/zfs-scripts/*zfs*.log > C (E) > > [root@dslustre21 check-mk-agent]# mv ../logwatch.cfg . mv: overwrite `./logwatch.cfg'? yes [root@dslustre21 check-mk-agent]# pwd /etc/check-mk-agent [root@dslustre21 check-mk-agent]# ll /etc/check-mk-agent/logwatch.cfg -rw-r--r-- 1 root root 2456 Jan 13 10:16 /etc/check-mk-agent/logwatch.cfg
Then I added the new 4 services via checkmk's WATO
#9 Updated by Gerard Bernabeu Altayo almost 5 years ago
There is clearly a bug for my script, the CRC file looks like this:
[root@dslustre21 ~]# cat /nexsanpool2/filelist-201601.crc axial-day-1-20160101.gz 364767830 axial-week-1-20160107.gz 3921114051 c51-day-1-20160101.gz 2900074934 charmonium-day-1-20160101.gz 2396880006 charmonium-week-1-20160107.gz 609896209 disco-day-1-20160101.gz 2245895316 disco-week-1-20160107.gz 1154236981 fourpluseight-day-1-20160101.gz 770351881 fourpluseight-week-1-20160107.gz 3778889209 heavylight-day-1-20160101.gz 1382710575 heavylight-week-1-20160107.gz 1064513988 hpqcd-day-1-20160101.gz 3915399009 hpqcd-week-1-20160107.gz 1012015823 mixbk-day-1-20160101.gz 860111476 mixbk-week-1-20160107.gz 3952952712 nonperturbativestaggered-day-1-20160101.gz 299362089 nonperturbativestaggered-week-1-20160107.gz 4190391641 ok-action-day-1-20160101.gz 3535741520 ok-action-week-1-20160107.gz 3533849934 rbcdwf-day-1-20160101.gz 3077220975 rbcdwf-week-1-20160107.gz 2595766803 rhqbbar-day-1-20160101.gz 2816352350 rhqbbar-week-1-20160107.gz 2393381739 c51-week-1-20160107.gz 259932587 charmonium-week-2-20160114.gz c51-week-2-20160114.gz disco-week-2-20160114.gz fourpluseight-week-2-20160114.gz mixbk-week-2-20160114.gz nonperturbativestaggered-week-2-20160114.gz rbcdwf-week-2-20160114.gz ok-action-week-2-20160114.gz hpqcd-week-2-20160114.gz axial-week-2-20160114.gz rhqbbar-week-2-20160114.gz heavylight-week-2-20160114.gz [root@dslustre21 ~]#
ie: it is missing all the CRCs, and new lines...
At least no new error was logged and all the snapshots happened as expected:
[root@dslustre21 zfs-scripts]# grep '(E)' *zfs*.log | grep 'Jan 1' daily-zfs-replica-snapshot.log:Jan 10 02:00:02 (E) Failed (256) to execute: /sbin/zfs send -v -i nexsanpool1/axial@day-1-20160101 nexsanpool1/axial@day-7-20160107 | /sbin/zfs recv nexsanpool2/axial@week-1-20160107 daily-zfs-replica-snapshot.log:Jan 11 02:00:45 (E) Failed (256) to execute: /sbin/zfs send -v -i nexsanpool1/axial@day-1-20160101 nexsanpool1/axial@day-7-20160107 | /sbin/zfs recv nexsanpool2/axial@week-1-20160107 delete-zfs-gzips.log:Wed Jan 13 10:04:35 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING delete-zfs-gzips.log:Wed Jan 13 10:26:20 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING delete-zfs-gzips.log:Wed Jan 13 10:29:16 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING delete-zfs-gzips.log:Wed Jan 13 10:38:27 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING delete-zfs-gzips.log:Wed Jan 13 10:39:26 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING delete-zfs-gzips.log:Wed Jan 13 10:39:46 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING delete-zfs-gzips.log:Wed Jan 13 11:18:46 CST 2016 (E) Could not locate a complete filelist-*.crc. STOPPING [root@dslustre21 zfs-scripts]#
If I look at the logs, I see:
[root@dslustre21 zfs-scripts]# tail daily-zfs-replica-snapshot.log Jan 15 02:05:03 (I) Making GZIP for rhqbbar Jan 15 02:05:03 (I) /nexsanpool2/rhqbbar-day-1-20160101.gz already exists Jan 15 02:05:03 (I) /nexsanpool2/rhqbbar-week-1-20160107.gz already exists sh: ecrc: command not found Jan 15 02:05:03 (I) Creating /nexsanpool2/rhqbbar-week-2-20160114.gz with: (/sbin/zfs send -i nexsanpool2/rhqbbar@week-1-20160107 nexsanpool2/rhqbbar@week-2-20160114 | gzip > /nexsanpool2/rhqbbar-week-2-20160114.gz && (echo -n 'rhqbbar-week-2-20160114.gz ' && ecrc /nexsanpool2/rhqbbar-week-2-20160114.gz | cut -d' ' -f 2 ) >> /nexsanpool2/filelist-201601.crc )& sh: ecrc: command not found sh: ecrc: command not found sh: ecrc: command not found sh: ecrc: command not found sh: ecrc: command not found [root@dslustre21 zfs-scripts]#
Soo, I need to add the full path for ecrc and this will probably work!
#10 Updated by Gerard Bernabeu Altayo almost 5 years ago
Done, it is now working fine:
Jan 19 10:42:40 (I) Making GZIP for rhqbbar Jan 19 10:42:40 (I) /nexsanpool2/rhqbbar-day-1-20160101.gz already exists Jan 19 10:42:40 (I) /nexsanpool2/rhqbbar-week-1-20160107.gz already exists Jan 19 10:42:40 (I) Creating /nexsanpool2/rhqbbar-week-2-20160114.gz in the background with: (/sbin/zfs send -i nexsanpool2/rhqbbar@week-1-20160107 nexsanpool2/rhqbbar@week-2-20160114 | /bin/gzip > /nexsanpool2/rhqbbar-week-2-20160114.gz && (echo -n 'rhqbbar-week-2-20160114.gz ' && /usr/local/bin/ecrc /nexsanpool2/rhqbbar-week-2-20160114.gz | cut -d' ' -f 2 ) >> /nexsanpool2/filelist-201601.crc )& [root@dslustre21 zfs-scripts]# cat /nexsanpool2/filelist-201601.crc axial-day-1-20160101.gz 364767830 axial-week-1-20160107.gz 3921114051 c51-day-1-20160101.gz 2900074934 charmonium-day-1-20160101.gz 2396880006 charmonium-week-1-20160107.gz 609896209 disco-day-1-20160101.gz 2245895316 disco-week-1-20160107.gz 1154236981 fourpluseight-day-1-20160101.gz 770351881 fourpluseight-week-1-20160107.gz 3778889209 heavylight-day-1-20160101.gz 1382710575 heavylight-week-1-20160107.gz 1064513988 hpqcd-day-1-20160101.gz 3915399009 hpqcd-week-1-20160107.gz 1012015823 mixbk-day-1-20160101.gz 860111476 mixbk-week-1-20160107.gz 3952952712 nonperturbativestaggered-day-1-20160101.gz 299362089 nonperturbativestaggered-week-1-20160107.gz 4190391641 ok-action-day-1-20160101.gz 3535741520 ok-action-week-1-20160107.gz 3533849934 rbcdwf-day-1-20160101.gz 3077220975 rbcdwf-week-1-20160107.gz 2595766803 rhqbbar-day-1-20160101.gz 2816352350 rhqbbar-week-1-20160107.gz 2393381739 c51-week-1-20160107.gz 259932587 charmonium-week-2-20160114.gz 218384547 c51-week-2-20160114.gz 2372418994 disco-week-2-20160114.gz 4010689085 fourpluseight-week-2-20160114.gz 1536640057 mixbk-week-2-20160114.gz 3358277476 nonperturbativestaggered-week-2-20160114.gz 2586593235 ok-action-week-2-20160114.gz 2669561960 rbcdwf-week-2-20160114.gz 115425591 [root@dslustre21 zfs-scripts]#
Also fixed the watchlog.cfg config to escape the (, now it looks like:
#Checks for ZFS backups /root/zfs-scripts/*zfs*.log C \(E\)
#11 Updated by Gerard Bernabeu Altayo over 4 years ago
The backup process still has some bug, in this case a race condition that causes the CRC to be written wrong:
[root@dslustre21 ~]# tail -12 /nexsanpool2/filelist-201601.crc
c51-week-4-20160128.gz charmonium-week-4-20160128.gz 1189101752
2019846639
disco-week-4-20160128.gz 2641314519
fourpluseight-week-4-20160128.gz 149965703
hpqcd-week-4-20160128.gz 1946370591
mixbk-week-4-20160128.gz 2044020069
ok-action-week-4-20160128.gz 3095743034
rbcdwf-week-4-20160128.gz 336871318
axial-week-4-20160128.gz 3756750631
rhqbbar-week-4-20160128.gz 604327362
heavylight-week-4-20160128.gz 2042313975
nonperturbativestaggered-week-4-20160128.gz 1802902312
[root@dslustre21 ~]#
[root@dslustre21 zfs-scripts]# git diff
diff --git a/replica-zfs-snapshot.pl b/replica-zfs-snapshot.pl
index 32f9103..38a7308 100755
--- a/replica-zfs-snapshot.pl
+++ b/replica-zfs-snapshot.pl
@@ -147,9 +147,9 @@ foreach my $dataset (@datasets) {
my $date=(split "-", $nexan2snaps[$i])[2];
my $crcfile="/nexsanpool2/filelist-".(substr $date, 0, 6).".crc";
if (grep /day-1-.+/, $nexan2snaps[$i]) {
- $cmd="($ZFS send nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i].gz ' && /usr/local/bin/ecrc $dest | cut -d' ' -f 2) >> $c
+ $cmd="(/bin/sleep 1; $ZFS send nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i].gz ' && /usr/local/bin/ecrc $dest | cut -d'
} else {
- $cmd="($ZFS send -i nexsanpool2/$dataset\@$nexan2snaps[$i-1] nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i].gz ' && /usr/
+ $cmd="(/bin/sleep 1; $ZFS send -i nexsanpool2/$dataset\@$nexan2snaps[$i-1] nexsanpool2/$dataset\@$nexan2snaps[$i] | /bin/gzip > $dest && (echo -n '$dataset-$nexan2snaps[$i]
}
&printLog("I","Creating $dest in the background with: $cmd");
system("$cmd");
[root@dslustre21 zfs-scripts]# git commit -m'adding sleep to minimize the race condition chances' -a
[master e7eb9bd] adding sleep to minimize the race condition chances
For now I will NOT fix the file as I want the checks to complain that something is wrong at the end of the month (in 3 days)
#12 Updated by Gerard Bernabeu Altayo over 4 years ago
Fixed the logic, now it works!
[root@dslustre21 ~]# zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
nexsanpool1 16.2T 7.74T 8.51T 47% 1.00x ONLINE -
nexsanpool2 16.2T 13.3T 2.98T 81% 1.00x ONLINE -
projectzfspool 7.25T 5.56T 1.69T 76% 1.00x ONLINE -
[root@dslustre21 ~]# zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
nexsanpool1/axial@day-1-20160201 0 - 13.1G -
nexsanpool1/c51@day-1-20160201 0 - 1.04G -
nexsanpool1/charmonium@day-1-20160201 0 - 615K -
nexsanpool1/disco@day-1-20160201 0 - 6.29G -
nexsanpool1/fourpluseight@day-1-20160201 0 - 22.1G -
nexsanpool1/heavylight@day-1-20160201 0 - 2.89T -
nexsanpool1/hpqcd@day-1-20160201 0 - 778G -
nexsanpool1/mixbk@day-1-20160201 0 - 268G -
nexsanpool1/nonperturbativestaggered@day-1-20160201 0 - 70.1G -
nexsanpool1/ok-action@day-1-20160201 0 - 8.79G -
nexsanpool1/rbcdwf@day-1-20160201 0 - 499G -
nexsanpool1/rhqbbar@day-1-20160201 0 - 654G -
nexsanpool2/axial@day-1-20160201 0 - 13.1G -
nexsanpool2/c51@day-1-20160201 0 - 1.04G -
nexsanpool2/charmonium@day-1-20160201 0 - 615K -
nexsanpool2/disco@day-1-20160201 0 - 6.29G -
nexsanpool2/fourpluseight@day-1-20160201 0 - 22.1G -
nexsanpool2/heavylight@day-1-20160201 0 - 2.89T -
nexsanpool2/hpqcd@day-1-20160201 0 - 778G -
nexsanpool2/mixbk@day-1-20160201 0 - 268G -
nexsanpool2/nonperturbativestaggered@day-1-20160201 0 - 70.1G -
nexsanpool2/ok-action@day-1-20160201 0 - 8.79G -
nexsanpool2/rbcdwf@day-1-20160201 0 - 499G -
nexsanpool2/rhqbbar@day-1-20160201 0 - 654G -
projectzfspool/projectzfs/axial@day-1-20160201 4.45M - 12.9G -
projectzfspool/projectzfs/c51@day-1-20160201 0 - 826M -
projectzfspool/projectzfs/charmonium@day-1-20160201 220K - 440K -
projectzfspool/projectzfs/disco@day-1-20160201 0 - 10.7G -
projectzfspool/projectzfs/fourpluseight@day-1-20160201 0 - 26.5G -
projectzfspool/projectzfs/heavylight@day-1-20160201 4.39M - 3.14T -
projectzfspool/projectzfs/hpqcd@day-1-20160201 0 - 506G -
projectzfspool/projectzfs/mixbk@day-1-20160201 0 - 487G -
projectzfspool/projectzfs/nonperturbativestaggered@day-1-20160201 0 - 152G -
projectzfspool/projectzfs/ok-action@day-1-20160201 0 - 12.6G -
projectzfspool/projectzfs/rbcdwf@day-1-20160201 0 - 572G -
projectzfspool/projectzfs/rhqbbar@day-1-20160201 0 - 694G -
[root@dslustre21 ~]# cd /root/zfs-scripts/
[root@dslustre21 zfs-scripts]# git log | head
commit bb56b57b5f071dda28bae35158f900011edb14f2
Author: root <root@dslustre21.fnal.gov>
Date: Tue Feb 2 14:40:43 2016 -0600
fixed a few bugscommit e7eb9bdde8683f05bb5d59f08bb8b53b174b9005
Author: root <root@dslustre21.fnal.gov>
Date: Fri Jan 29 13:53:49 2016 -0600
[root@dslustre21 zfs-scripts]# git show bb56b57b5f071dda28bae35158f900011edb14f2
commit bb56b57b5f071dda28bae35158f900011edb14f2
Author: root <root@dslustre21.fnal.gov>
Date: Tue Feb 2 14:40:43 2016 -0600
fixed a few bugsdiff --git a/first-of-month.pl b/first-of-month.pl
index 7d43a26..404f86c 100755
--- a/first-of-month.pl
+++ b/first-of-month.pl@ -2,9 +2,9 @
#
# This script is meant to run on the 2nd of the month to initialize backups for a next month, once all gzips have been generated and the gzips are good:
# 1. Continue only if last_month_day-1 and current_month_day-1 exist in projectzfs and all GZIPs (5 per experiment) are there# 2. Destroy all zpools&replicas from nexsanpool1 and nexsanpool2# 3. Copy day-1 from projectzfs to nexsanpool1 and from nexsanpool1 to nexsanpool2
-# 4. Remove all last month's snapshots from projecdtzfs
# 2. Copy day-1 incremental from projectzfs to nexsanpool1
# 3. Copy day-1 incremental from nexsanpool1 to nexsanpool2
+# 4. Remove all last month's snapshots from projecdtzfs, nexsanpool1 and nexsanpool2
use Date::Manip;@ -35,8 +35,10 @ while(<>)
}
-my @datasetstodestroy;
+my @datasetstoreplicate;
# 1. Continue only if last_month_day-1 and current_month_day-1 exist in projectzfs and all GZIPs (5 per experiment) are there
# 2. Copy day-1 incremental from projectzfs to nexsanpool1
DATASET: foreach $dataset (datasets) { -f 2`;
&printLog("I","Checking snapshots for $dataset");
my @projectzfspoolsnaps=split "\n", `$ZFS list -H -r -t snapshot -o name projectzfspool/projectzfs/$dataset | cut -d@ -70,37 +72,17 @ DATASET: foreach $dataset (@datasets) {
}
if ( (scalar missingsnaps == 0) && grep(/day-1-$this_month_day1/,@projectzfspoolsnaps) && grep(/day-1-$last_month_day1/,@projectzfspoolsnaps) ) {
- # 2. Destroy all dataset snapshots from nexsanpool1
- my @nexan1snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool1/$dataset | cut -d f 2`;
foreach my $snap (nexsan1snaps) {
- $cmd="$ZFS destroy -v nexsanpool1/$dataset\$snap";
- &printLog("I",$cmd);
- system($cmd);
- if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n");
- }
- }
# 3. Create the first snapshot of the month: day-1 from projectzfs to nexsanpool1. This used to be a big simple send!
$cmd="$ZFS send v -i projectzfspool/projectzfs/$dataset\day-".UnixDate($last_month_last_day, "%d")."-".UnixDate($last_month_last_day, "%Y%m%d")." projectzfspool/projectzfs/$d
&printLog("I",$cmd);
system($cmd);
if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n NOT cleaning up nexsanpool2/$dataset\n nor removing projectzfspool/projectzfs/$dataset");
+ printLog ("E","Failed ($?) to execute: $cmd\n NOT cleaning up nexsanpool(1|2)/$dataset\n nor removing projectzfspool/projectzfs/$dataset");
next DATASET;
}
else {
- #If we succeed we add to the list of datasets to destroy from nexsanpool2
- push @datasetstodestroy, $dataset;
- }
- # 4. Remove all last month's snapshots from projecdtzfs
- @toremove=grep(/day-.+-$last_month.+/, @projectzfspoolsnaps);
- foreach my $snap (@toremove) {
- $cmd="$ZFS destroy -v projectzfspool/projectzfs/$dataset\$snap";
&printLog("I",$cmd);
- system($cmd);
- if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n");
- }
+ #If we succeed we add to the list of datasets to destroy from projectzfs and nexsanpool2
+ push @datasetstoreplicate, $dataset;
}} else {
@ -109,12 +91,32 @ DATASET: foreach $dataset (@datasets) {}+# 3. Copy day-1 incremental from nexsanpool1 to nexsanpool2
+my @datasetstodestroy;
+foreach $dataset (@datasetstoreplicate) {
+ #Copy from nexsanpool1 to nexsanpool2
+ $cmd="$ZFS send v -i nexsanpool1/$dataset\@day".UnixDate($last_month_last_day, "%d")."-".UnixDate($last_month_last_day, "%Y%m%d")." nexsanpool1/$dataset\@day-1-$this_month_da
+ #$cmd="$ZFS send -v nexsanpool1/$dataset\@day-1-$this_month_day1 | $ZFS recv nexsanpool2/$dataset &";
+ &printLog("I",$cmd);
+ system($cmd);
+ if ($? != 0) {
+ printLog ("E","Failed ($?) to execute: $cmd\n NOT cleaning up nexsanpool(1|2)/$dataset\n nor removing projectzfspool/projectzfs/$dataset");
+ next DATASET;
+ }
+ else {
+ #If we succeed we add to the list of datasets to destroy from projectzfs and nexsanpool2
+ push @datasetstodestroy, $dataset;
+ }
+}
+# 4. Remove all last month's snapshots from projecdtzfs, nexsanpool1 and nexsanpool2
foreach $dataset (@datasetstodestroy) {
- my nexan2snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool2/$dataset | cut -d f 2`; foreach my $snap (nexsan2snaps) { -f 2`;
+ # Destroy all last month's dataset snapshots from nexsanpool2
+ my @nexsan2snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool2/$dataset | cut -d
+ toremove=grep(/.+-$last_month.+/, @nexsan2snaps);$snap";
+ foreach my $snap (@toremove) {
$cmd="$ZFS destroy -v nexsanpool2/$dataset\
&printLog("I",$cmd);
system($cmd);@ -123,13 +125,28 @ foreach $dataset (@datasetstodestroy) {
}
}
- #Copy from nexsanpool1 to nexsanpool2
- $cmd="$ZFS send -v -i projectzfspool/projectzfs/$dataset\day-".UnixDate($last_month_last_day, "%d")."-".UnixDate($last_month_last_day, "%Y%m%d")." projectzfspool/projectzfs/$d -f 2`;
- #$cmd="$ZFS send -v nexsanpool1/$dataset\@day-1-$this_month_day1 | $ZFS recv nexsanpool2/$dataset &";
- &printLog("I",$cmd);
- system($cmd);
- if ($? != 0) {
- printLog ("E","Failed ($?) to execute: $cmd\n");
+ # Destroy all last month's dataset snapshots from nexsanpool1
+ my @nexsan1snaps=split "\n", `$ZFS list -H -r -t snapshot -o name nexsanpool1/$dataset | cut -d
+ toremove=grep(/day-.+-$last_month.+/, @nexsan1snaps);$snap";
+ foreach my $snap (@toremove) {
+ $cmd="$ZFS destroy -v nexsanpool1/$dataset\
+ &printLog("I",$cmd);
+ system($cmd);
+ if ($? != 0) {
+ printLog ("E","Failed ($?) to execute: $cmd\n");
+ }
+ }
# Remove all last month's snapshots from projectzfs
+ my projectzfspoolsnaps=split "\n", `$ZFS list -H -r -t snapshot -o name projectzfspool/projectzfs/$dataset | cut -d -f 2`;
+ toremove=grep(/day-.+-$last_month.+/, @projectzfspoolsnaps);$snap";
+ foreach my $snap (@toremove) {
+ $cmd="$ZFS destroy -v projectzfspool/projectzfs/$dataset\
+ &printLog("I",$cmd);
+ system($cmd);
+ if ($? != 0) {
+ printLog ("E","Failed ($?) to execute: $cmd\n");
+ }
}
}
[root@dslustre21 zfs-scripts]##13 Updated by Gerard Bernabeu Altayo over 4 years ago
- % Done changed from 0 to 90
Now the gzip removal does not work because it sees that the files are not on tape yet:
[root@dslustre21 ~]# /root/zfs-scripts/delete-gzips.sh ls: /pnfs/lqcd/backups/lqcdproj_backups/project_backup_*/*/filelist-201601.crc: No such file or directory dirname: missing operand Try `dirname --help' for more information. Tue Feb 2 17:14:17 CST 2016 (E) Could not locate a complete filelist-*.crc on ENSTORE. STOPPING [root@dslustre21 ~]# echo $? 98 [root@dslustre21 ~]#
I am adding the cleanup script in a crontab entry, to run on the 4th from dslustre21:
#Monthtly GZIP cleanup. Runs on the 4th of the month to make sure all GZIPs have been removed, only works if tape copies succeeded and after checking checksum is OK. Copy to tape is not automated here! Copy to tape runs from [root@lqcdsrm ~]# /home/gerard1/backups_to_tape.sh. 00 12 4 * * (/root/zfs-scripts/delete-gzips.sh >> /root/zfs-scripts/delete-zfs-gzips.log 2>&1)
Also now automatically running the copy to tape from lqcdsrm crontab now:
# Montthly copy of the GZIPs to tape, GZIP cleanup done from dslustre21 00 16 1 * * (/home/gerard1/backups_to_tape.sh >> /home/gerard1/backups_to_tape.log 2>&1)
I am running it manually now (from a screen), and next month we'll see if it works! If it does, I should tweak the script to have 'standard' greppable output and add a checkmk sensor for it...
#14 Updated by Gerard Bernabeu Altayo over 4 years ago
All files where transferred successfully to tape:
[root@lqcdsrm ~]# grep 'Overall rate' /var/log/lqcd-backup/encp_bkp.2016-02-02--172433.log | wc -l 73
Tomorrow the script that does the removals will do all necessary checks and remove the files, I tried with just an echo on the RM and it worked fine, so I removed the echo:
[root@dslustre21 ~]# /root/zfs-scripts/delete-gzips.sh Wed Feb 3 12:06:50 CST 2016 (I) There are 72 /nexsanpool2/filelist-201601.crc files to be considered for removal Wed Feb 3 12:06:50 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-day-1-20160101.gz vs axial-day-1-20160101.gz Wed Feb 3 12:06:52 CST 2016 (I) Removing. Checksums are 364767830 vs 364767830 Wed Feb 3 12:06:52 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-1-20160107.gz vs axial-week-1-20160107.gz Wed Feb 3 12:06:53 CST 2016 (I) Removing. Checksums are 3921114051 vs 3921114051 Wed Feb 3 12:06:53 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-day-1-20160101.gz vs c51-day-1-20160101.gz Wed Feb 3 12:06:53 CST 2016 (I) Removing. Checksums are 2900074934 vs 2900074934 Wed Feb 3 12:06:53 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-day-1-20160101.gz vs charmonium-day-1-20160101.gz Wed Feb 3 12:06:53 CST 2016 (I) Removing. Checksums are 2396880006 vs 2396880006 Wed Feb 3 12:06:53 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-1-20160107.gz vs charmonium-week-1-20160107.gz Wed Feb 3 12:06:54 CST 2016 (I) Removing. Checksums are 609896209 vs 609896209 Wed Feb 3 12:06:54 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-day-1-20160101.gz vs disco-day-1-20160101.gz Wed Feb 3 12:06:54 CST 2016 (I) Removing. Checksums are 2245895316 vs 2245895316 Wed Feb 3 12:06:54 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-1-20160107.gz vs disco-week-1-20160107.gz Wed Feb 3 12:06:54 CST 2016 (I) Removing. Checksums are 1154236981 vs 1154236981 Wed Feb 3 12:06:54 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-day-1-20160101.gz vs fourpluseight-day-1-20160101.gz Wed Feb 3 12:06:55 CST 2016 (I) Removing. Checksums are 770351881 vs 770351881 Wed Feb 3 12:06:55 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-1-20160107.gz vs fourpluseight-week-1-20160107.gz Wed Feb 3 12:06:55 CST 2016 (I) Removing. Checksums are 3778889209 vs 3778889209 Wed Feb 3 12:06:55 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-day-1-20160101.gz vs heavylight-day-1-20160101.gz Wed Feb 3 12:06:55 CST 2016 (I) Removing. Checksums are 1382710575 vs 1382710575 Wed Feb 3 12:06:55 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-1-20160107.gz vs heavylight-week-1-20160107.gz Wed Feb 3 12:06:56 CST 2016 (I) Removing. Checksums are 1064513988 vs 1064513988 Wed Feb 3 12:06:56 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-day-1-20160101.gz vs hpqcd-day-1-20160101.gz Wed Feb 3 12:06:56 CST 2016 (I) Removing. Checksums are 3915399009 vs 3915399009 Wed Feb 3 12:06:56 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-1-20160107.gz vs hpqcd-week-1-20160107.gz Wed Feb 3 12:06:56 CST 2016 (I) Removing. Checksums are 1012015823 vs 1012015823 Wed Feb 3 12:06:56 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-day-1-20160101.gz vs mixbk-day-1-20160101.gz Wed Feb 3 12:06:57 CST 2016 (I) Removing. Checksums are 860111476 vs 860111476 Wed Feb 3 12:06:57 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-1-20160107.gz vs mixbk-week-1-20160107.gz Wed Feb 3 12:06:57 CST 2016 (I) Removing. Checksums are 3952952712 vs 3952952712 Wed Feb 3 12:06:57 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-day-1-20160101.gz vs nonperturbativestaggered-day-1-20160101.gz Wed Feb 3 12:06:57 CST 2016 (I) Removing. Checksums are 299362089 vs 299362089 Wed Feb 3 12:06:57 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-1-20160107.gz vs nonperturbativestaggered-week-1-20160107.gz Wed Feb 3 12:06:58 CST 2016 (I) Removing. Checksums are 4190391641 vs 4190391641 Wed Feb 3 12:06:58 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-day-1-20160101.gz vs ok-action-day-1-20160101.gz Wed Feb 3 12:06:58 CST 2016 (I) Removing. Checksums are 3535741520 vs 3535741520 Wed Feb 3 12:06:58 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-1-20160107.gz vs ok-action-week-1-20160107.gz Wed Feb 3 12:06:58 CST 2016 (I) Removing. Checksums are 3533849934 vs 3533849934 Wed Feb 3 12:06:58 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-day-1-20160101.gz vs rbcdwf-day-1-20160101.gz Wed Feb 3 12:06:59 CST 2016 (I) Removing. Checksums are 3077220975 vs 3077220975 Wed Feb 3 12:06:59 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-1-20160107.gz vs rbcdwf-week-1-20160107.gz Wed Feb 3 12:06:59 CST 2016 (I) Removing. Checksums are 2595766803 vs 2595766803 Wed Feb 3 12:06:59 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-day-1-20160101.gz vs rhqbbar-day-1-20160101.gz Wed Feb 3 12:06:59 CST 2016 (I) Removing. Checksums are 2816352350 vs 2816352350 Wed Feb 3 12:06:59 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-1-20160107.gz vs rhqbbar-week-1-20160107.gz Wed Feb 3 12:07:00 CST 2016 (I) Removing. Checksums are 2393381739 vs 2393381739 Wed Feb 3 12:07:00 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-1-20160107.gz vs c51-week-1-20160107.gz Wed Feb 3 12:07:00 CST 2016 (I) Removing. Checksums are 259932587 vs 259932587 Wed Feb 3 12:07:00 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-2-20160114.gz vs charmonium-week-2-20160114.gz Wed Feb 3 12:07:00 CST 2016 (I) Removing. Checksums are 218384547 vs 218384547 Wed Feb 3 12:07:00 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-2-20160114.gz vs c51-week-2-20160114.gz Wed Feb 3 12:07:01 CST 2016 (I) Removing. Checksums are 2372418994 vs 2372418994 Wed Feb 3 12:07:01 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-2-20160114.gz vs disco-week-2-20160114.gz Wed Feb 3 12:07:01 CST 2016 (I) Removing. Checksums are 4010689085 vs 4010689085 Wed Feb 3 12:07:01 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-2-20160114.gz vs fourpluseight-week-2-20160114.gz Wed Feb 3 12:07:01 CST 2016 (I) Removing. Checksums are 1536640057 vs 1536640057 Wed Feb 3 12:07:01 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-2-20160114.gz vs mixbk-week-2-20160114.gz Wed Feb 3 12:07:02 CST 2016 (I) Removing. Checksums are 3358277476 vs 3358277476 Wed Feb 3 12:07:02 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-2-20160114.gz vs nonperturbativestaggered-week-2-20160114.gz Wed Feb 3 12:07:02 CST 2016 (I) Removing. Checksums are 2586593235 vs 2586593235 Wed Feb 3 12:07:02 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-2-20160114.gz vs ok-action-week-2-20160114.gz Wed Feb 3 12:07:03 CST 2016 (I) Removing. Checksums are 2669561960 vs 2669561960 Wed Feb 3 12:07:03 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-2-20160114.gz vs rbcdwf-week-2-20160114.gz Wed Feb 3 12:07:03 CST 2016 (I) Removing. Checksums are 115425591 vs 115425591 Wed Feb 3 12:07:03 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-2-20160114.gz vs hpqcd-week-2-20160114.gz Wed Feb 3 12:07:03 CST 2016 (I) Removing. Checksums are 3935780579 vs 3935780579 Wed Feb 3 12:07:03 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-2-20160114.gz vs axial-week-2-20160114.gz Wed Feb 3 12:07:04 CST 2016 (I) Removing. Checksums are 4279856720 vs 4279856720 Wed Feb 3 12:07:04 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-2-20160114.gz vs rhqbbar-week-2-20160114.gz Wed Feb 3 12:07:04 CST 2016 (I) Removing. Checksums are 266787443 vs 266787443 Wed Feb 3 12:07:04 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-2-20160114.gz vs heavylight-week-2-20160114.gz Wed Feb 3 12:07:04 CST 2016 (I) Removing. Checksums are 68087258 vs 68087258 Wed Feb 3 12:07:04 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-3-20160121.gz vs charmonium-week-3-20160121.gz Wed Feb 3 12:07:05 CST 2016 (I) Removing. Checksums are 3201975380 vs 3201975380 Wed Feb 3 12:07:05 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-3-20160121.gz vs c51-week-3-20160121.gz Wed Feb 3 12:07:05 CST 2016 (I) Removing. Checksums are 1260142674 vs 1260142674 Wed Feb 3 12:07:05 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-3-20160121.gz vs disco-week-3-20160121.gz Wed Feb 3 12:07:05 CST 2016 (I) Removing. Checksums are 2831499310 vs 2831499310 Wed Feb 3 12:07:05 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-3-20160121.gz vs fourpluseight-week-3-20160121.gz Wed Feb 3 12:07:06 CST 2016 (I) Removing. Checksums are 225002564 vs 225002564 Wed Feb 3 12:07:06 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-3-20160121.gz vs hpqcd-week-3-20160121.gz Wed Feb 3 12:07:06 CST 2016 (I) Removing. Checksums are 2258386864 vs 2258386864 Wed Feb 3 12:07:06 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-3-20160121.gz vs mixbk-week-3-20160121.gz Wed Feb 3 12:07:06 CST 2016 (I) Removing. Checksums are 2474000592 vs 2474000592 Wed Feb 3 12:07:06 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-3-20160121.gz vs ok-action-week-3-20160121.gz Wed Feb 3 12:07:07 CST 2016 (I) Removing. Checksums are 154747309 vs 154747309 Wed Feb 3 12:07:07 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-3-20160121.gz vs nonperturbativestaggered-week-3-20160121.gz Wed Feb 3 12:07:07 CST 2016 (I) Removing. Checksums are 388507985 vs 388507985 Wed Feb 3 12:07:07 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-3-20160121.gz vs rbcdwf-week-3-20160121.gz Wed Feb 3 12:07:07 CST 2016 (I) Removing. Checksums are 382615602 vs 382615602 Wed Feb 3 12:07:07 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-3-20160121.gz vs axial-week-3-20160121.gz Wed Feb 3 12:07:08 CST 2016 (I) Removing. Checksums are 3882272373 vs 3882272373 Wed Feb 3 12:07:08 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-3-20160121.gz vs rhqbbar-week-3-20160121.gz Wed Feb 3 12:07:08 CST 2016 (I) Removing. Checksums are 1606419459 vs 1606419459 Wed Feb 3 12:07:08 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-3-20160121.gz vs heavylight-week-3-20160121.gz Wed Feb 3 12:07:08 CST 2016 (I) Removing. Checksums are 787402680 vs 787402680 Wed Feb 3 12:07:08 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-4-20160128.gz vs c51-week-4-20160128.gz Wed Feb 3 12:07:09 CST 2016 (I) Removing. Checksums are 1189101752 vs 1189101752 Wed Feb 3 12:07:09 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-4-20160128.gz vs charmonium-week-4-20160128.gz Wed Feb 3 12:07:09 CST 2016 (I) Removing. Checksums are 2019846639 vs 2019846639 Wed Feb 3 12:07:09 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-4-20160128.gz vs disco-week-4-20160128.gz Wed Feb 3 12:07:09 CST 2016 (I) Removing. Checksums are 2641314519 vs 2641314519 Wed Feb 3 12:07:09 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-4-20160128.gz vs fourpluseight-week-4-20160128.gz Wed Feb 3 12:07:10 CST 2016 (I) Removing. Checksums are 149965703 vs 149965703 Wed Feb 3 12:07:10 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-4-20160128.gz vs hpqcd-week-4-20160128.gz Wed Feb 3 12:07:10 CST 2016 (I) Removing. Checksums are 1946370591 vs 1946370591 Wed Feb 3 12:07:10 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-4-20160128.gz vs mixbk-week-4-20160128.gz Wed Feb 3 12:07:10 CST 2016 (I) Removing. Checksums are 2044020069 vs 2044020069 Wed Feb 3 12:07:10 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-4-20160128.gz vs ok-action-week-4-20160128.gz Wed Feb 3 12:07:11 CST 2016 (I) Removing. Checksums are 3095743034 vs 3095743034 Wed Feb 3 12:07:11 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-4-20160128.gz vs rbcdwf-week-4-20160128.gz Wed Feb 3 12:07:11 CST 2016 (I) Removing. Checksums are 336871318 vs 336871318 Wed Feb 3 12:07:11 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-4-20160128.gz vs axial-week-4-20160128.gz Wed Feb 3 12:07:11 CST 2016 (I) Removing. Checksums are 3756750631 vs 3756750631 Wed Feb 3 12:07:11 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-4-20160128.gz vs rhqbbar-week-4-20160128.gz Wed Feb 3 12:07:12 CST 2016 (I) Removing. Checksums are 604327362 vs 604327362 Wed Feb 3 12:07:12 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-4-20160128.gz vs heavylight-week-4-20160128.gz Wed Feb 3 12:07:12 CST 2016 (I) Removing. Checksums are 2042313975 vs 2042313975 Wed Feb 3 12:07:12 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-4-20160128.gz vs nonperturbativestaggered-week-4-20160128.gz Wed Feb 3 12:07:13 CST 2016 (I) Removing. Checksums are 1802902312 vs 1802902312 Wed Feb 3 12:07:13 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/c51-week-5-20160131.gz vs c51-week-5-20160131.gz Wed Feb 3 12:07:13 CST 2016 (I) Removing. Checksums are 4139860148 vs 4139860148 Wed Feb 3 12:07:13 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/disco-week-5-20160131.gz vs disco-week-5-20160131.gz Wed Feb 3 12:07:13 CST 2016 (I) Removing. Checksums are 3311092616 vs 3311092616 Wed Feb 3 12:07:13 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/charmonium-week-5-20160131.gz vs charmonium-week-5-20160131.gz Wed Feb 3 12:07:14 CST 2016 (I) Removing. Checksums are 2309531583 vs 2309531583 Wed Feb 3 12:07:14 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/fourpluseight-week-5-20160131.gz vs fourpluseight-week-5-20160131.gz Wed Feb 3 12:07:14 CST 2016 (I) Removing. Checksums are 124929347 vs 124929347 Wed Feb 3 12:07:14 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/mixbk-week-5-20160131.gz vs mixbk-week-5-20160131.gz Wed Feb 3 12:07:14 CST 2016 (I) Removing. Checksums are 2890088644 vs 2890088644 Wed Feb 3 12:07:14 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/nonperturbativestaggered-week-5-20160131.gz vs nonperturbativestaggered-week-5-20160131.gz Wed Feb 3 12:07:15 CST 2016 (I) Removing. Checksums are 913591135 vs 913591135 Wed Feb 3 12:07:15 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/ok-action-week-5-20160131.gz vs ok-action-week-5-20160131.gz Wed Feb 3 12:07:15 CST 2016 (I) Removing. Checksums are 2775204969 vs 2775204969 Wed Feb 3 12:07:15 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rbcdwf-week-5-20160131.gz vs rbcdwf-week-5-20160131.gz Wed Feb 3 12:07:15 CST 2016 (I) Removing. Checksums are 112671935 vs 112671935 Wed Feb 3 12:07:15 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/axial-week-5-20160131.gz vs axial-week-5-20160131.gz Wed Feb 3 12:07:16 CST 2016 (I) Removing. Checksums are 2065256347 vs 2065256347 Wed Feb 3 12:07:16 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/hpqcd-week-5-20160131.gz vs hpqcd-week-5-20160131.gz Wed Feb 3 12:07:16 CST 2016 (I) Removing. Checksums are 695194048 vs 695194048 Wed Feb 3 12:07:16 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/heavylight-week-5-20160131.gz vs heavylight-week-5-20160131.gz Wed Feb 3 12:07:16 CST 2016 (I) Removing. Checksums are 2430992215 vs 2430992215 Wed Feb 3 12:07:16 CST 2016 (I) Checking on /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2016-02-02/rhqbbar-week-5-20160131.gz vs rhqbbar-week-5-20160131.gz Wed Feb 3 12:07:17 CST 2016 (I) Removing. Checksums are 346937683 vs 346937683 rm -f /nexsanpool2/filelist-201601.crc /nexsanpool2/axial-day-1-20160101.gz /nexsanpool2/axial-week-1-20160107.gz /nexsanpool2/c51-day-1-20160101.gz /nexsanpool2/charmonium-day-1-20160101.gz /nexsanpool2/charmonium-week-1-20160107.gz /nexsanpool2/disco-day-1-20160101.gz /nexsanpool2/disco-week-1-20160107.gz /nexsanpool2/fourpluseight-day-1-20160101.gz /nexsanpool2/fourpluseight-week-1-20160107.gz /nexsanpool2/heavylight-day-1-20160101.gz /nexsanpool2/heavylight-week-1-20160107.gz /nexsanpool2/hpqcd-day-1-20160101.gz /nexsanpool2/hpqcd-week-1-20160107.gz /nexsanpool2/mixbk-day-1-20160101.gz /nexsanpool2/mixbk-week-1-20160107.gz /nexsanpool2/nonperturbativestaggered-day-1-20160101.gz /nexsanpool2/nonperturbativestaggered-week-1-20160107.gz /nexsanpool2/ok-action-day-1-20160101.gz /nexsanpool2/ok-action-week-1-20160107.gz /nexsanpool2/rbcdwf-day-1-20160101.gz /nexsanpool2/rbcdwf-week-1-20160107.gz /nexsanpool2/rhqbbar-day-1-20160101.gz /nexsanpool2/rhqbbar-week-1-20160107.gz /nexsanpool2/c51-week-1-20160107.gz /nexsanpool2/charmonium-week-2-20160114.gz /nexsanpool2/c51-week-2-20160114.gz /nexsanpool2/disco-week-2-20160114.gz /nexsanpool2/fourpluseight-week-2-20160114.gz /nexsanpool2/mixbk-week-2-20160114.gz /nexsanpool2/nonperturbativestaggered-week-2-20160114.gz /nexsanpool2/ok-action-week-2-20160114.gz /nexsanpool2/rbcdwf-week-2-20160114.gz /nexsanpool2/hpqcd-week-2-20160114.gz /nexsanpool2/axial-week-2-20160114.gz /nexsanpool2/rhqbbar-week-2-20160114.gz /nexsanpool2/heavylight-week-2-20160114.gz /nexsanpool2/charmonium-week-3-20160121.gz /nexsanpool2/c51-week-3-20160121.gz /nexsanpool2/disco-week-3-20160121.gz /nexsanpool2/fourpluseight-week-3-20160121.gz /nexsanpool2/hpqcd-week-3-20160121.gz /nexsanpool2/mixbk-week-3-20160121.gz /nexsanpool2/ok-action-week-3-20160121.gz /nexsanpool2/nonperturbativestaggered-week-3-20160121.gz /nexsanpool2/rbcdwf-week-3-20160121.gz /nexsanpool2/axial-week-3-20160121.gz /nexsanpool2/rhqbbar-week-3-20160121.gz /nexsanpool2/heavylight-week-3-20160121.gz /nexsanpool2/c51-week-4-20160128.gz /nexsanpool2/charmonium-week-4-20160128.gz /nexsanpool2/disco-week-4-20160128.gz /nexsanpool2/fourpluseight-week-4-20160128.gz /nexsanpool2/hpqcd-week-4-20160128.gz /nexsanpool2/mixbk-week-4-20160128.gz /nexsanpool2/ok-action-week-4-20160128.gz /nexsanpool2/rbcdwf-week-4-20160128.gz /nexsanpool2/axial-week-4-20160128.gz /nexsanpool2/rhqbbar-week-4-20160128.gz /nexsanpool2/heavylight-week-4-20160128.gz /nexsanpool2/nonperturbativestaggered-week-4-20160128.gz /nexsanpool2/c51-week-5-20160131.gz /nexsanpool2/disco-week-5-20160131.gz /nexsanpool2/charmonium-week-5-20160131.gz /nexsanpool2/fourpluseight-week-5-20160131.gz /nexsanpool2/mixbk-week-5-20160131.gz /nexsanpool2/nonperturbativestaggered-week-5-20160131.gz /nexsanpool2/ok-action-week-5-20160131.gz /nexsanpool2/rbcdwf-week-5-20160131.gz /nexsanpool2/axial-week-5-20160131.gz /nexsanpool2/hpqcd-week-5-20160131.gz /nexsanpool2/heavylight-week-5-20160131.gz /nexsanpool2/rhqbbar-week-5-20160131.gz [root@dslustre21 ~]#
I had to do some fixing to the script but now it's commited in GIT and tomorrow the files should be gone, which is very good because space is tight!
[root@dslustre21 ~]# zpool list NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT nexsanpool1 16.2T 7.74T 8.51T 47% 1.00x ONLINE - nexsanpool2 16.2T 15.1T 1.14T 92% 1.00x ONLINE - projectzfspool 7.25T 5.56T 1.69T 76% 1.00x ONLINE - [root@dslustre21 ~]#
#15 Updated by Gerard Bernabeu Altayo over 4 years ago
The only pending thing now is to watch this for 1 month, if it all works then we're good to go!
#16 Updated by Gerard Bernabeu Altayo over 4 years ago
- Status changed from New to Resolved
it worked just fine for the February backups, closing ticket.