Project

General

Profile

Task #11112

HPC - dsluster21 backups

Added by Gerard Bernabeu Altayo almost 5 years ago. Updated over 4 years ago.

Status:
Resolved
Priority:
Normal
Start date:
12/09/2015
Due date:
% Done:

0%

Estimated time:
Duration:

Description

logbook for creating backups, this task needs to be automated

Photo on 12-10-15 at 2.24 PM.jpg (129 KB) Photo on 12-10-15 at 2.24 PM.jpg Gerard Bernabeu Altayo, 12/10/2015 02:24 PM

History

#1 Updated by Gerard Bernabeu Altayo almost 5 years ago

Need to create increments between days 6 and 7 from:

[root@dslustre21 ~]# cd /root/zfs-scripts/
[root@dslustre21 zfs-scripts]# zfs list
NAME                                                 USED  AVAIL  REFER  MOUNTPOINT
nexsanpool1                                         5.07T  5.58T   480K  /nexsanpool1
nexsanpool1/axial                                   12.1G  5.58T  12.0G  /nexsanpool1/axial
nexsanpool1/c51                                      922M  5.58T   922M  /nexsanpool1/c51
nexsanpool1/charmonium                               352K  5.58T   272K  /nexsanpool1/charmonium
nexsanpool1/disco                                   6.29G  5.58T  6.29G  /nexsanpool1/disco
nexsanpool1/fourpluseight                           20.5G  5.58T  20.5G  /nexsanpool1/fourpluseight
nexsanpool1/heavylight                              2.76T  5.58T  2.76T  /nexsanpool1/heavylight
nexsanpool1/hpqcd                                    777G  5.58T   777G  /nexsanpool1/hpqcd
nexsanpool1/mixbk                                    268G  5.58T   268G  /nexsanpool1/mixbk
nexsanpool1/nonperturbativestaggered                39.5G  5.58T  39.5G  /nexsanpool1/nonperturbativestaggered
nexsanpool1/ok-action                               8.79G  5.58T  8.79G  /nexsanpool1/ok-action
nexsanpool1/rbcdwf                                   499G  5.58T   499G  /nexsanpool1/rbcdwf
nexsanpool1/rhqbbar                                  729G  5.58T   729G  /nexsanpool1/rhqbbar
nexsanpool2                                         8.56T  2.10T  3.49T  /nexsanpool2
nexsanpool2/axial                                   12.0G  2.10T  12.0G  /nexsanpool2/axial
nexsanpool2/c51                                      922M  2.10T   922M  /nexsanpool2/c51
nexsanpool2/charmonium                               272K  2.10T   272K  /nexsanpool2/charmonium
nexsanpool2/disco                                   6.29G  2.10T  6.29G  /nexsanpool2/disco
nexsanpool2/fourpluseight                           20.5G  2.10T  20.5G  /nexsanpool2/fourpluseight
nexsanpool2/heavylight                              2.76T  2.10T  2.76T  /nexsanpool2/heavylight
nexsanpool2/hpqcd                                    777G  2.10T   777G  /nexsanpool2/hpqcd
nexsanpool2/mixbk                                    268G  2.10T   268G  /nexsanpool2/mixbk
nexsanpool2/nonperturbativestaggered                39.5G  2.10T  39.5G  /nexsanpool2/nonperturbativestaggered
nexsanpool2/ok-action                               8.79G  2.10T  8.79G  /nexsanpool2/ok-action
nexsanpool2/rbcdwf                                   499G  2.10T   499G  /nexsanpool2/rbcdwf
nexsanpool2/rhqbbar                                  729G  2.10T   729G  /nexsanpool2/rhqbbar
projectzfspool                                      5.54T  1.60T   144K  none
projectzfspool/projectzfs                           5.54T  1.60T   232K  /projectzfs
projectzfspool/projectzfs/axial                     11.9G  1.60T  11.9G  /projectzfs/axial
projectzfspool/projectzfs/c51                        747M  1.60T   747M  /projectzfs/c51
projectzfspool/projectzfs/charmonium                 136K  1.60T   136K  /projectzfs/charmonium
projectzfspool/projectzfs/disco                     10.7G  1.60T  10.7G  /projectzfs/disco
projectzfspool/projectzfs/fourpluseight             25.5G  1.60T  25.5G  /projectzfs/fourpluseight
projectzfspool/projectzfs/heavylight                3.03T  1.60T  3.03T  /projectzfs/heavylight
projectzfspool/projectzfs/hpqcd                      505G  1.60T   505G  /projectzfs/hpqcd
projectzfspool/projectzfs/mixbk                      487G  1.60T   487G  /projectzfs/mixbk
projectzfspool/projectzfs/nonperturbativestaggered   122G  1.60T   122G  /projectzfs/nonperturbativestaggered
projectzfspool/projectzfs/ok-action                 12.6G  1.60T  12.6G  /projectzfs/ok-action
projectzfspool/projectzfs/rbcdwf                     572G  1.60T   572G  /projectzfs/rbcdwf
projectzfspool/projectzfs/rhqbbar                    816G  1.60T   816G  /projectzfs/rhqbbar
[root@dslustre21 zfs-scripts]# 
[root@dslustre21 zfs-scripts]# zfs list -t snapshot
NAME                                                                USED  AVAIL  REFER  MOUNTPOINT
nexsanpool1/axial@day-1-20151201                                   16.0K      -  12.0G  -
nexsanpool1/axial@day-2-20151202                                   16.0K      -  12.0G  -
nexsanpool1/axial@day-3-20151203                                   16.0K      -  12.0G  -
nexsanpool1/axial@day-4-20151204                                   16.0K      -  12.0G  -
nexsanpool1/axial@day-5-20151205                                   16.0K      -  12.0G  -
nexsanpool1/axial@day-6-20151206                                       0      -  12.0G  -
nexsanpool1/c51@day-1-20151201                                     16.0K      -   922M  -
nexsanpool1/c51@day-2-20151202                                     16.0K      -   922M  -
nexsanpool1/c51@day-3-20151203                                     16.0K      -   922M  -
nexsanpool1/c51@day-4-20151204                                     16.0K      -   922M  -
nexsanpool1/c51@day-5-20151205                                     16.0K      -   922M  -
nexsanpool1/c51@day-6-20151206                                         0      -   922M  -
nexsanpool1/charmonium@day-1-20151201                              16.0K      -   272K  -
nexsanpool1/charmonium@day-2-20151202                              16.0K      -   272K  -
nexsanpool1/charmonium@day-3-20151203                              16.0K      -   272K  -
nexsanpool1/charmonium@day-4-20151204                              16.0K      -   272K  -
nexsanpool1/charmonium@day-5-20151205                              16.0K      -   272K  -
nexsanpool1/charmonium@day-6-20151206                                  0      -   272K  -
nexsanpool1/disco@day-1-20151201                                   16.0K      -  6.29G  -
nexsanpool1/disco@day-2-20151202                                   16.0K      -  6.29G  -
nexsanpool1/disco@day-3-20151203                                   16.0K      -  6.29G  -
nexsanpool1/disco@day-4-20151204                                   16.0K      -  6.29G  -
nexsanpool1/disco@day-5-20151205                                   16.0K      -  6.29G  -
nexsanpool1/disco@day-6-20151206                                       0      -  6.29G  -
nexsanpool1/fourpluseight@day-1-20151201                           16.0K      -  20.5G  -
nexsanpool1/fourpluseight@day-2-20151202                           16.0K      -  20.5G  -
nexsanpool1/fourpluseight@day-3-20151203                           16.0K      -  20.5G  -
nexsanpool1/fourpluseight@day-4-20151204                           16.0K      -  20.5G  -
nexsanpool1/fourpluseight@day-5-20151205                           16.0K      -  20.5G  -
nexsanpool1/fourpluseight@day-6-20151206                               0      -  20.5G  -
nexsanpool1/heavylight@day-1-20151201                               464K      -  2.76T  -
nexsanpool1/heavylight@day-2-20151202                               464K      -  2.76T  -
nexsanpool1/heavylight@day-3-20151203                              13.4M      -  2.76T  -
nexsanpool1/heavylight@day-4-20151204                              12.0M      -  2.76T  -
nexsanpool1/heavylight@day-5-20151205                              13.0M      -  2.76T  -
nexsanpool1/heavylight@day-6-20151206                                  0      -  2.76T  -
nexsanpool1/hpqcd@day-1-20151201                                   16.0K      -   777G  -
nexsanpool1/hpqcd@day-2-20151202                                   16.0K      -   777G  -
nexsanpool1/hpqcd@day-3-20151203                                   16.0K      -   777G  -
nexsanpool1/hpqcd@day-4-20151204                                   16.0K      -   777G  -
nexsanpool1/hpqcd@day-5-20151205                                   16.0K      -   777G  -
nexsanpool1/hpqcd@day-6-20151206                                       0      -   777G  -
nexsanpool1/mixbk@day-1-20151201                                   16.0K      -   268G  -
nexsanpool1/mixbk@day-2-20151202                                   16.0K      -   268G  -
nexsanpool1/mixbk@day-3-20151203                                   16.0K      -   268G  -
nexsanpool1/mixbk@day-4-20151204                                   16.0K      -   268G  -
nexsanpool1/mixbk@day-5-20151205                                   16.0K      -   268G  -
nexsanpool1/mixbk@day-6-20151206                                       0      -   268G  -
nexsanpool1/nonperturbativestaggered@day-1-20151201                16.0K      -  39.5G  -
nexsanpool1/nonperturbativestaggered@day-2-20151202                16.0K      -  39.5G  -
nexsanpool1/nonperturbativestaggered@day-3-20151203                16.0K      -  39.5G  -
nexsanpool1/nonperturbativestaggered@day-4-20151204                16.0K      -  39.5G  -
nexsanpool1/nonperturbativestaggered@day-5-20151205                16.0K      -  39.5G  -
nexsanpool1/nonperturbativestaggered@day-6-20151206                    0      -  39.5G  -
nexsanpool1/ok-action@day-1-20151201                               16.0K      -  8.79G  -
nexsanpool1/ok-action@day-2-20151202                               16.0K      -  8.79G  -
nexsanpool1/ok-action@day-3-20151203                               16.0K      -  8.79G  -
nexsanpool1/ok-action@day-4-20151204                               16.0K      -  8.79G  -
nexsanpool1/ok-action@day-5-20151205                               16.0K      -  8.79G  -
nexsanpool1/ok-action@day-6-20151206                                   0      -  8.79G  -
nexsanpool1/rbcdwf@day-1-20151201                                  16.0K      -   499G  -
nexsanpool1/rbcdwf@day-2-20151202                                  16.0K      -   499G  -
nexsanpool1/rbcdwf@day-3-20151203                                  16.0K      -   499G  -
nexsanpool1/rbcdwf@day-4-20151204                                  16.0K      -   499G  -
nexsanpool1/rbcdwf@day-5-20151205                                  16.0K      -   499G  -
nexsanpool1/rbcdwf@day-6-20151206                                      0      -   499G  -
nexsanpool1/rhqbbar@day-1-20151201                                 16.0K      -   729G  -
nexsanpool1/rhqbbar@day-2-20151202                                 16.0K      -   729G  -
nexsanpool1/rhqbbar@day-3-20151203                                 16.0K      -   729G  -
nexsanpool1/rhqbbar@day-4-20151204                                 16.0K      -   729G  -
nexsanpool1/rhqbbar@day-5-20151205                                 16.0K      -   729G  -
nexsanpool1/rhqbbar@day-6-20151206                                     0      -   729G  -
nexsanpool2/axial@day-1-20151201                                       0      -  12.0G  -
nexsanpool2/c51@day-1-20151201                                         0      -   922M  -
nexsanpool2/charmonium@day-1-20151201                                  0      -   272K  -
nexsanpool2/disco@day-1-20151201                                       0      -  6.29G  -
nexsanpool2/fourpluseight@day-1-20151201                               0      -  20.5G  -
nexsanpool2/heavylight@day-1-20151201                                  0      -  2.76T  -
nexsanpool2/hpqcd@day-1-20151201                                       0      -   777G  -
nexsanpool2/mixbk@day-1-20151201                                       0      -   268G  -
nexsanpool2/nonperturbativestaggered@day-1-20151201                    0      -  39.5G  -
nexsanpool2/ok-action@day-1-20151201                                   0      -  8.79G  -
nexsanpool2/rbcdwf@day-1-20151201                                      0      -   499G  -
nexsanpool2/rhqbbar@day-1-20151201                                     0      -   729G  -
projectzfspool/projectzfs/axial@day-1-20151201                         0      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20151202                         0      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20151203                         0      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20151204                         0      -  11.9G  -
projectzfspool/projectzfs/axial@day-5-20151205                         0      -  11.9G  -
projectzfspool/projectzfs/axial@day-6-20151206                         0      -  11.9G  -
projectzfspool/projectzfs/axial@day-7-20151207                         0      -  11.9G  -
projectzfspool/projectzfs/axial@day-8-20151208                         0      -  11.9G  -
projectzfspool/projectzfs/c51@day-1-20151201                           0      -   747M  -
projectzfspool/projectzfs/c51@day-2-20151202                           0      -   747M  -
projectzfspool/projectzfs/c51@day-3-20151203                           0      -   747M  -
projectzfspool/projectzfs/c51@day-4-20151204                           0      -   747M  -
projectzfspool/projectzfs/c51@day-5-20151205                           0      -   747M  -
projectzfspool/projectzfs/c51@day-6-20151206                           0      -   747M  -
projectzfspool/projectzfs/c51@day-7-20151207                           0      -   747M  -
projectzfspool/projectzfs/c51@day-8-20151208                           0      -   747M  -
projectzfspool/projectzfs/charmonium@day-1-20151201                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20151202                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20151203                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20151204                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-5-20151205                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-6-20151206                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-7-20151207                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-8-20151208                    0      -   136K  -
projectzfspool/projectzfs/disco@day-1-20151201                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-2-20151202                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-3-20151203                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-4-20151204                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-5-20151205                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-6-20151206                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-7-20151207                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-8-20151208                         0      -  10.7G  -
projectzfspool/projectzfs/fourpluseight@day-1-20151201                 0      -  25.5G  -
projectzfspool/projectzfs/fourpluseight@day-2-20151202                 0      -  25.5G  -
projectzfspool/projectzfs/fourpluseight@day-3-20151203                 0      -  25.5G  -
projectzfspool/projectzfs/fourpluseight@day-4-20151204                 0      -  25.5G  -
projectzfspool/projectzfs/fourpluseight@day-5-20151205                 0      -  25.5G  -
projectzfspool/projectzfs/fourpluseight@day-6-20151206                 0      -  25.5G  -
projectzfspool/projectzfs/fourpluseight@day-7-20151207                 0      -  25.5G  -
projectzfspool/projectzfs/fourpluseight@day-8-20151208                 0      -  25.5G  -
projectzfspool/projectzfs/heavylight@day-1-20151201                 232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-2-20151202                 232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-3-20151203                7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-4-20151204                6.75M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-5-20151205                7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-6-20151206                19.5M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-7-20151207                2.50G      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-8-20151208                 248K      -  3.03T  -
projectzfspool/projectzfs/hpqcd@day-1-20151201                         0      -   505G  -
projectzfspool/projectzfs/hpqcd@day-2-20151202                         0      -   505G  -
projectzfspool/projectzfs/hpqcd@day-3-20151203                         0      -   505G  -
projectzfspool/projectzfs/hpqcd@day-4-20151204                         0      -   505G  -
projectzfspool/projectzfs/hpqcd@day-5-20151205                         0      -   505G  -
projectzfspool/projectzfs/hpqcd@day-6-20151206                         0      -   505G  -
projectzfspool/projectzfs/hpqcd@day-7-20151207                         0      -   505G  -
projectzfspool/projectzfs/hpqcd@day-8-20151208                         0      -   505G  -
projectzfspool/projectzfs/mixbk@day-1-20151201                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-2-20151202                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-3-20151203                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-4-20151204                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-5-20151205                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-6-20151206                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-7-20151207                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-8-20151208                         0      -   487G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-1-20151201      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-2-20151202      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-3-20151203      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-4-20151204      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-5-20151205      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-6-20151206      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-7-20151207      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-8-20151208      0      -   122G  -
projectzfspool/projectzfs/ok-action@day-1-20151201                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-2-20151202                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-3-20151203                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-4-20151204                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-5-20151205                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-6-20151206                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-7-20151207                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-8-20151208                     0      -  12.6G  -
projectzfspool/projectzfs/rbcdwf@day-1-20151201                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-2-20151202                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-3-20151203                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-4-20151204                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-5-20151205                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-6-20151206                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-7-20151207                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-8-20151208                        0      -   572G  -
projectzfspool/projectzfs/rhqbbar@day-1-20151201                       0      -   816G  -
projectzfspool/projectzfs/rhqbbar@day-2-20151202                       0      -   816G  -
projectzfspool/projectzfs/rhqbbar@day-3-20151203                       0      -   816G  -
projectzfspool/projectzfs/rhqbbar@day-4-20151204                       0      -   816G  -
projectzfspool/projectzfs/rhqbbar@day-5-20151205                       0      -   816G  -
projectzfspool/projectzfs/rhqbbar@day-6-20151206                       0      -   816G  -
projectzfspool/projectzfs/rhqbbar@day-7-20151207                       0      -   816G  -
projectzfspool/projectzfs/rhqbbar@day-8-20151208                       0      -   816G  -
[root@dslustre21 zfs-scripts]# for i in `cat dataset-list-reduced`; do zfs send -v -i projectzfspool/projectzfs/$i@day-6-20151206 projectzfspool/projectzfs/$i@day-7-20151207 | zfs recv nexsanpool1/$i@day-7-20151207; done
send from @day-6-20151206 to projectzfspool/projectzfs/axial@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/charmonium@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/c51@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/disco@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/fourpluseight@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/heavylight@day-7-20151207 estimated size is 5.19G
total estimated size is 5.19G
TIME        SENT   SNAPSHOT
11:53:32   34.4M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:33    116M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:34    200M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:35    287M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:36    373M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:37    458M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:38    541M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:39    616M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:40    703M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:41    783M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:42    872M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:43    952M   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:44   1.00G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:45   1.08G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:46   1.08G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:47   1.08G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:48   1.09G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:49   1.22G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:50   1.37G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:51   1.48G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:52   1.61G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:53   1.77G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:54   1.93G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:55   2.08G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:56   2.24G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:57   2.39G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:58   2.55G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:53:59   2.73G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:00   2.89G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:01   3.07G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:02   3.26G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:03   3.44G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:04   3.59G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:05   3.79G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:06   3.98G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:07   4.16G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:08   4.30G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:09   4.43G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:10   4.52G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:11   4.61G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:12   4.77G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:13   4.97G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:14   5.11G   projectzfspool/projectzfs/heavylight@day-7-20151207
11:54:15   5.16G   projectzfspool/projectzfs/heavylight@day-7-20151207
send from @day-6-20151206 to projectzfspool/projectzfs/hpqcd@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/mixbk@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/nonperturbativestaggered@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/ok-action@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/rbcdwf@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
send from @day-6-20151206 to projectzfspool/projectzfs/rhqbbar@day-7-20151207 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
[root@dslustre21 zfs-scripts]# 

#2 Updated by Gerard Bernabeu Altayo almost 5 years ago

Now I just run it for one more day, in the end it's just this line:

[root@dslustre21 zfs-scripts]# for i in `cat dataset-list-reduced`; do zfs send -v -i projectzfspool/projectzfs/$i@day-7-20151207 projectzfspool/projectzfs/$i@day-8-20151208 | zfs recv nexsanpool1/$i@day-8-20151208; done

#3 Updated by Gerard Bernabeu Altayo almost 5 years ago

We can check if it's properly done by running:

[root@dslustre21 zfs-scripts]# ./decision-noemail.pl 
REPORT:    Day snapshots are 8 on project, 8 on pool1 and 1 on pool2
REFERENCE: Day snapshots should be 1-30 on project, 1-30 on pool1 and 1 on pool2
REPORT:    Week snapshots are 0 on pool2
REFERENCE: Week snapshots should be 1-4 on pool2

$> zfs list -t snapshot
NAME                                                                USED  AVAIL  REFER  MOUNTPOINT
nexsanpool1/axial@day-1-20151201                                   16.0K      -  12.0G  -
nexsanpool1/axial@day-2-20151202                                   16.0K      -  12.0G  -
nexsanpool1/axial@day-3-20151203                                   16.0K      -  12.0G  -
....

See the first line, numbers should match....

Now let's go for the weekly snapshot backup. Not to overload the live ZPOOL we'll do it from nexan1, which is the 'cold replica' disk:

[root@dslustre21 zfs-scripts]# screen 

[root@dslustre21 zfs-scripts]# for i in `cat dataset-list-reduced`; do (zfs send -v -i nexsanpool1/$i@day-1-20151201 nexsanpool1/$i@day-7-20151207 | zfs recv nexsanpool2/$i@week-1-20151207); done

#4 Updated by Gerard Bernabeu Altayo almost 5 years ago

Now we prepare for the monthly backups, which we always do from day1 and the 4 weekly backups, generating the GZIPs.

This takes up to a couple days!!!

[root@dslustre21 zfs-scripts]# screen
[root@dslustre21 zfs-scripts]# zfs list -t snapshot | grep day-1 | head -1
nexsanpool1/axial@day-1-20151201                                   16.0K      -  12.0G  -
[root@dslustre21 zfs-scripts]# cat generate-gzips.txt 
# day
for i in `cat dataset-list-reduced`;do (zfs send nexsanpool2/$i@day-1-20150713 | gzip > /nexsanpool2/$i-day-1-20150713.gz &);done

# week
for i in `cat dataset-list-reduced`;do (zfs send -i nexsanpool2/$i@day-1-20150727 nexsanpool2/$i@week-1-20150802 | gzip > /nexsanpool2/$i-week-1-20150802.gz &);done
[root@dslustre21 zfs-scripts]# for i in `cat dataset-list-reduced`;do (zfs send nexsanpool2/$i@day-1-20151201 | gzip > /nexsanpool2/$i-day-1-20151201.gz &);done

#In parallel we can start moving the weekly snapshots:
[root@dslustre21 zfs-scripts]# for i in `cat dataset-list-reduced`;do (zfs send -i nexsanpool2/$i@day-1-20151201 nexsanpool2/$i@week-1-20151207 | gzip > /nexsanpool2/$i-week-1-20151207.gz &);done

Next thing we will do is go to lqcdsrm and move the data from that mountpoint on dslustre21 (via infiniband) to Enstore.

#5 Updated by Gerard Bernabeu Altayo almost 5 years ago

To to the tape backups:

[root@lqcdsrm ~]# less /home/amitoj/backup/bin/bkw.sh

Each month one needs to change one of the first lines 'pnfsdir=/pnfs/lqcd/backups/lqcdproj_backups/project_backup_A/2015-12-01'. On 2015-12-01 you'll put the NOVEMBER'15 backups and so on, the day just means when the backup was performed and it takes 'last month's' backup gzips.

The script is called as:

[root@lqcdsrm ~]# screen
[root@lqcdsrm ~]# /home/amitoj/backup/bin/bkw.sh /nexsanpool2/list

Where 'list' is on dslustre21:/nexsanpool2 and comes from 'ls *.gz > list'.

#6 Updated by Gerard Bernabeu Altayo almost 5 years ago

I am working on a script that does all the missing daily backups (the 1st step I had to learn), seems to work:

[root@dslustre21 zfs.gba]# echo charmonium | ./replica-zfs-snapshot.sh
Dec 22 14:15:21 (I) Snapshot date tag is 22

Dec 22 14:15:21 (I) Reading dataset list from stdin ..
Dec 22 14:15:21 (I) Replicating snapshots for charmonium
Dec 22 14:15:22 (I) Creating snapshot nexsanpool1/charmonium@day-17-20151217
zfs send -v -i projectzfspool/projectzfs/charmonium@day-16-20151216 projectzfspool/projectzfs/charmonium@day-17-20151217 | zfs recv nexsanpool1/charmonium@day-17-20151217
Dec 22 14:15:22 (I) Creating snapshot nexsanpool1/charmonium@day-18-20151218
zfs send -v -i projectzfspool/projectzfs/charmonium@day-17-20151217 projectzfspool/projectzfs/charmonium@day-18-20151218 | zfs recv nexsanpool1/charmonium@day-18-20151218
Dec 22 14:15:22 (I) Creating snapshot nexsanpool1/charmonium@day-19-20151219
zfs send -v -i projectzfspool/projectzfs/charmonium@day-18-20151218 projectzfspool/projectzfs/charmonium@day-19-20151219 | zfs recv nexsanpool1/charmonium@day-19-20151219
Dec 22 14:15:22 (I) Creating snapshot nexsanpool1/charmonium@day-20-20151220
zfs send -v -i projectzfspool/projectzfs/charmonium@day-19-20151219 projectzfspool/projectzfs/charmonium@day-20-20151220 | zfs recv nexsanpool1/charmonium@day-20-20151220
Dec 22 14:15:22 (I) Creating snapshot nexsanpool1/charmonium@day-21-20151221
zfs send -v -i projectzfspool/projectzfs/charmonium@day-20-20151220 projectzfspool/projectzfs/charmonium@day-21-20151221 | zfs recv nexsanpool1/charmonium@day-21-20151221
[root@dslustre21 zfs.gba]# zfs send -v -i projectzfspool/projectzfs/charmonium@day-16-20151216 projectzfspool/projectzfs/charmonium@day-17-20151217 | zfs recv nexsanpool1/charmonium@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/charmonium@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
[root@dslustre21 zfs.gba]# echo charmonium | ./replica-zfs-snapshot.sh
Dec 22 14:17:10 (I) Snapshot date tag is 22

Dec 22 14:17:10 (I) Reading dataset list from stdin ..
Dec 22 14:17:10 (I) Replicating snapshots for charmonium
Dec 22 14:17:10 (I) Creating snapshot nexsanpool1/charmonium@day-18-20151218
zfs send -v -i projectzfspool/projectzfs/charmonium@day-17-20151217 projectzfspool/projectzfs/charmonium@day-18-20151218 | zfs recv nexsanpool1/charmonium@day-18-20151218
Dec 22 14:17:10 (I) Creating snapshot nexsanpool1/charmonium@day-19-20151219
zfs send -v -i projectzfspool/projectzfs/charmonium@day-18-20151218 projectzfspool/projectzfs/charmonium@day-19-20151219 | zfs recv nexsanpool1/charmonium@day-19-20151219
Dec 22 14:17:10 (I) Creating snapshot nexsanpool1/charmonium@day-20-20151220
zfs send -v -i projectzfspool/projectzfs/charmonium@day-19-20151219 projectzfspool/projectzfs/charmonium@day-20-20151220 | zfs recv nexsanpool1/charmonium@day-20-20151220
Dec 22 14:17:10 (I) Creating snapshot nexsanpool1/charmonium@day-21-20151221
zfs send -v -i projectzfspool/projectzfs/charmonium@day-20-20151220 projectzfspool/projectzfs/charmonium@day-21-20151221 | zfs recv nexsanpool1/charmonium@day-21-20151221
[root@dslustre21 zfs.gba]# 

It is safe to run anywhere.

#7 Updated by Gerard Bernabeu Altayo almost 5 years ago

It works and is safe to run as often as desired:

[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep charmonium
nexsanpool1/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool1/charmonium@day-2-20151202                               16.0K      -   272K  -
nexsanpool1/charmonium@day-3-20151203                               16.0K      -   272K  -
nexsanpool1/charmonium@day-4-20151204                               16.0K      -   272K  -
nexsanpool1/charmonium@day-5-20151205                               16.0K      -   272K  -
nexsanpool1/charmonium@day-6-20151206                               16.0K      -   272K  -
nexsanpool1/charmonium@day-7-20151207                               16.0K      -   272K  -
nexsanpool1/charmonium@day-8-20151208                               16.0K      -   272K  -
nexsanpool1/charmonium@day-9-20151209                               16.0K      -   272K  -
nexsanpool1/charmonium@day-10-20151210                              16.0K      -   272K  -
nexsanpool1/charmonium@day-11-20151211                              16.0K      -   272K  -
nexsanpool1/charmonium@day-12-20151212                              16.0K      -   272K  -
nexsanpool1/charmonium@day-13-20151213                              16.0K      -   272K  -
nexsanpool1/charmonium@day-14-20151214                              16.0K      -   272K  -
nexsanpool1/charmonium@day-15-20151215                              16.0K      -   272K  -
nexsanpool1/charmonium@day-16-20151216                              16.0K      -   272K  -
nexsanpool1/charmonium@day-17-20151217                                  0      -   272K  -
nexsanpool2/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool2/charmonium@week-1-20151207                              16.0K      -   272K  -
nexsanpool2/charmonium@week-2-20151214                                  0      -   272K  -
projectzfspool/projectzfs/charmonium@day-1-20151201                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20151202                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20151203                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20151204                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-5-20151205                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-6-20151206                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-7-20151207                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-8-20151208                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-9-20151209                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-10-20151210                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-11-20151211                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-12-20151212                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-13-20151213                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-14-20151214                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-15-20151215                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-16-20151216                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-17-20151217                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-18-20151218                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-19-20151219                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-20-20151220                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-21-20151221                    0      -   136K  -
[root@dslustre21 zfs.gba]# echo charmonium | ./replica-zfs-snapshot.sh
Dec 22 14:25:40 (I) Snapshot date tag is 22

Dec 22 14:25:40 (I) Reading dataset list from stdin ..
Dec 22 14:25:40 (I) Replicating snapshots for charmonium
Dec 22 14:25:40 (I) Creating snapshot nexsanpool1/charmonium@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/charmonium@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:25:40 (I) Creating snapshot nexsanpool1/charmonium@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/charmonium@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:25:41 (I) Creating snapshot nexsanpool1/charmonium@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/charmonium@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:25:41 (I) Creating snapshot nexsanpool1/charmonium@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/charmonium@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
[root@dslustre21 zfs.gba]# echo charmonium | ./replica-zfs-snapshot.sh
Dec 22 14:25:55 (I) Snapshot date tag is 22

Dec 22 14:25:55 (I) Reading dataset list from stdin ..
Dec 22 14:25:55 (I) Replicating snapshots for charmonium
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep charmonium
nexsanpool1/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool1/charmonium@day-2-20151202                               16.0K      -   272K  -
nexsanpool1/charmonium@day-3-20151203                               16.0K      -   272K  -
nexsanpool1/charmonium@day-4-20151204                               16.0K      -   272K  -
nexsanpool1/charmonium@day-5-20151205                               16.0K      -   272K  -
nexsanpool1/charmonium@day-6-20151206                               16.0K      -   272K  -
nexsanpool1/charmonium@day-7-20151207                               16.0K      -   272K  -
nexsanpool1/charmonium@day-8-20151208                               16.0K      -   272K  -
nexsanpool1/charmonium@day-9-20151209                               16.0K      -   272K  -
nexsanpool1/charmonium@day-10-20151210                              16.0K      -   272K  -
nexsanpool1/charmonium@day-11-20151211                              16.0K      -   272K  -
nexsanpool1/charmonium@day-12-20151212                              16.0K      -   272K  -
nexsanpool1/charmonium@day-13-20151213                              16.0K      -   272K  -
nexsanpool1/charmonium@day-14-20151214                              16.0K      -   272K  -
nexsanpool1/charmonium@day-15-20151215                              16.0K      -   272K  -
nexsanpool1/charmonium@day-16-20151216                              16.0K      -   272K  -
nexsanpool1/charmonium@day-17-20151217                              16.0K      -   272K  -
nexsanpool1/charmonium@day-18-20151218                              16.0K      -   272K  -
nexsanpool1/charmonium@day-19-20151219                              16.0K      -   272K  -
nexsanpool1/charmonium@day-20-20151220                              16.0K      -   272K  -
nexsanpool1/charmonium@day-21-20151221                                  0      -   272K  -
nexsanpool2/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool2/charmonium@week-1-20151207                              16.0K      -   272K  -
nexsanpool2/charmonium@week-2-20151214                                  0      -   272K  -
projectzfspool/projectzfs/charmonium@day-1-20151201                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20151202                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20151203                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20151204                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-5-20151205                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-6-20151206                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-7-20151207                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-8-20151208                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-9-20151209                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-10-20151210                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-11-20151211                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-12-20151212                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-13-20151213                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-14-20151214                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-15-20151215                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-16-20151216                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-17-20151217                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-18-20151218                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-19-20151219                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-20-20151220                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-21-20151221                    0      -   136K  -
[root@dslustre21 zfs.gba]# 

Still need to solve how to create the day 1, but we did not cover this 'corner case'.

Running for all:

[root@dslustre21 zfs.gba]# cat /root/zfs-scripts/dataset-list-reduced | ./replica-zfs-snapshot.sh
Dec 22 14:27:21 (I) Snapshot date tag is 22

Dec 22 14:27:21 (I) Reading dataset list from stdin ..
Dec 22 14:27:21 (I) Replicating snapshots for axial
Dec 22 14:27:21 (I) Creating snapshot nexsanpool1/axial@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/axial@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:22 (I) Creating snapshot nexsanpool1/axial@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/axial@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:23 (I) Creating snapshot nexsanpool1/axial@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/axial@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:23 (I) Creating snapshot nexsanpool1/axial@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/axial@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:24 (I) Creating snapshot nexsanpool1/axial@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/axial@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:24 (I) Replicating snapshots for charmonium
Dec 22 14:27:25 (I) Replicating snapshots for c51
Dec 22 14:27:25 (I) Creating snapshot nexsanpool1/c51@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/c51@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:25 (I) Creating snapshot nexsanpool1/c51@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/c51@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:26 (I) Creating snapshot nexsanpool1/c51@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/c51@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:26 (I) Creating snapshot nexsanpool1/c51@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/c51@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:27 (I) Creating snapshot nexsanpool1/c51@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/c51@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:27 (I) Replicating snapshots for disco
Dec 22 14:27:27 (I) Creating snapshot nexsanpool1/disco@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/disco@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:28 (I) Creating snapshot nexsanpool1/disco@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/disco@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:29 (I) Creating snapshot nexsanpool1/disco@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/disco@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:29 (I) Creating snapshot nexsanpool1/disco@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/disco@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:30 (I) Creating snapshot nexsanpool1/disco@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/disco@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:30 (I) Replicating snapshots for fourpluseight
Dec 22 14:27:30 (I) Creating snapshot nexsanpool1/fourpluseight@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/fourpluseight@day-17-20151217 estimated size is 728M
total estimated size is 728M
TIME        SENT   SNAPSHOT
14:27:31   35.4M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:32   67.8M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:33    112M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:34    149M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:35    184M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:36    213M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:37    259M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:38    278M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:39    369M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:40    454M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:41    496M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:42    603M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
14:27:43    779M   projectzfspool/projectzfs/fourpluseight@day-17-20151217
Dec 22 14:27:46 (I) Creating snapshot nexsanpool1/fourpluseight@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/fourpluseight@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:46 (I) Creating snapshot nexsanpool1/fourpluseight@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/fourpluseight@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:47 (I) Creating snapshot nexsanpool1/fourpluseight@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/fourpluseight@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:48 (I) Creating snapshot nexsanpool1/fourpluseight@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/fourpluseight@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:48 (I) Replicating snapshots for heavylight
Dec 22 14:27:48 (I) Creating snapshot nexsanpool1/heavylight@day-11-20151211
WARNING: could not send projectzfspool/projectzfs/heavylight@day-11-20151211:
incremental source (projectzfspool/projectzfs/heavylight@day-16-20151216) is not earlier than it
total estimated size is 0
cannot receive: failed to read from stream
Dec 22 14:27:48 (I) Creating snapshot nexsanpool1/heavylight@day-13-20151213
send from @day-11-20151211 to projectzfspool/projectzfs/heavylight@day-13-20151213 estimated size is 840M
total estimated size is 840M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-13-20151213': Broken pipe
Dec 22 14:27:48 (I) Creating snapshot nexsanpool1/heavylight@day-17-20151217
send from @day-13-20151213 to projectzfspool/projectzfs/heavylight@day-17-20151217 estimated size is 6.65G
total estimated size is 6.65G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-17-20151217': Broken pipe
Dec 22 14:27:49 (I) Creating snapshot nexsanpool1/heavylight@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/heavylight@day-18-20151218 estimated size is 5.86G
total estimated size is 5.86G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-18-20151218': Broken pipe
Dec 22 14:27:49 (I) Creating snapshot nexsanpool1/heavylight@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/heavylight@day-19-20151219 estimated size is 4.87G
total estimated size is 4.87G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-19-20151219': Broken pipe
Dec 22 14:27:49 (I) Creating snapshot nexsanpool1/heavylight@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/heavylight@day-20-20151220 estimated size is 4.89G
total estimated size is 4.89G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-20-20151220': Broken pipe
Dec 22 14:27:49 (I) Creating snapshot nexsanpool1/heavylight@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/heavylight@day-21-20151221 estimated size is 5.13G
total estimated size is 5.13G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-21-20151221': Broken pipe
Dec 22 14:27:49 (I) Replicating snapshots for hpqcd
Dec 22 14:27:49 (I) Creating snapshot nexsanpool1/hpqcd@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/hpqcd@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:50 (I) Creating snapshot nexsanpool1/hpqcd@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/hpqcd@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:51 (I) Creating snapshot nexsanpool1/hpqcd@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/hpqcd@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:51 (I) Creating snapshot nexsanpool1/hpqcd@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/hpqcd@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:52 (I) Creating snapshot nexsanpool1/hpqcd@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/hpqcd@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:52 (I) Replicating snapshots for mixbk
Dec 22 14:27:52 (I) Creating snapshot nexsanpool1/mixbk@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/mixbk@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:53 (I) Creating snapshot nexsanpool1/mixbk@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/mixbk@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:54 (I) Creating snapshot nexsanpool1/mixbk@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/mixbk@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:54 (I) Creating snapshot nexsanpool1/mixbk@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/mixbk@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:55 (I) Creating snapshot nexsanpool1/mixbk@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/mixbk@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:55 (I) Replicating snapshots for nonperturbativestaggered
Dec 22 14:27:55 (I) Creating snapshot nexsanpool1/nonperturbativestaggered@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/nonperturbativestaggered@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:56 (I) Creating snapshot nexsanpool1/nonperturbativestaggered@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/nonperturbativestaggered@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:56 (I) Creating snapshot nexsanpool1/nonperturbativestaggered@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/nonperturbativestaggered@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:57 (I) Creating snapshot nexsanpool1/nonperturbativestaggered@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/nonperturbativestaggered@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:58 (I) Creating snapshot nexsanpool1/nonperturbativestaggered@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/nonperturbativestaggered@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:58 (I) Replicating snapshots for ok-action
Dec 22 14:27:58 (I) Creating snapshot nexsanpool1/ok-action@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/ok-action@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:59 (I) Creating snapshot nexsanpool1/ok-action@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/ok-action@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:27:59 (I) Creating snapshot nexsanpool1/ok-action@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/ok-action@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:00 (I) Creating snapshot nexsanpool1/ok-action@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/ok-action@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:00 (I) Creating snapshot nexsanpool1/ok-action@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/ok-action@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:01 (I) Replicating snapshots for rbcdwf
Dec 22 14:28:01 (I) Creating snapshot nexsanpool1/rbcdwf@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/rbcdwf@day-17-20151217 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:02 (I) Creating snapshot nexsanpool1/rbcdwf@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/rbcdwf@day-18-20151218 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:02 (I) Creating snapshot nexsanpool1/rbcdwf@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/rbcdwf@day-19-20151219 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:03 (I) Creating snapshot nexsanpool1/rbcdwf@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/rbcdwf@day-20-20151220 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:03 (I) Creating snapshot nexsanpool1/rbcdwf@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/rbcdwf@day-21-20151221 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Dec 22 14:28:04 (I) Replicating snapshots for rhqbbar
Dec 22 14:28:04 (I) Creating snapshot nexsanpool1/rhqbbar@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/rhqbbar@day-17-20151217 estimated size is 15.7G
total estimated size is 15.7G
TIME        SENT   SNAPSHOT
14:28:05   32.7M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:06   79.4M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:07    200M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:08    293M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:09    408M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:10    512M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:11    626M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:12    750M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:13    874M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:14    998M   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:15   1.08G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:16   1.20G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:17   1.32G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:18   1.42G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:19   1.52G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:20   1.62G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:21   1.71G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:22   1.82G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:23   1.93G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:24   2.02G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:25   2.11G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:26   2.22G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:27   2.33G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:28   2.46G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:29   2.57G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:30   2.69G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:31   2.81G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:32   2.93G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:33   3.05G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:34   3.17G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:35   3.29G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:36   3.40G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:37   3.51G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:38   3.62G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:39   3.74G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:40   3.85G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:41   3.92G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:42   4.04G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:43   4.15G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:44   4.29G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:45   4.49G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:46   4.61G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:47   4.73G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:48   4.85G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:49   4.96G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:50   5.07G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:51   5.19G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:52   5.30G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:53   5.41G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:54   5.51G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:55   5.63G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:56   5.74G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:57   5.83G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:58   5.93G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:28:59   6.04G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:00   6.15G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:01   6.26G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:02   6.38G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:03   6.50G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:04   6.59G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:05   6.60G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:06   6.66G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:07   6.78G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:08   6.89G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:09   7.00G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:10   7.12G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:11   7.21G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:12   7.33G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:13   7.45G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:14   7.55G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:15   7.66G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:16   7.77G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:17   7.88G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:18   7.99G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:19   8.09G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:20   8.20G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:21   8.32G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:22   8.43G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:23   8.54G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:24   8.66G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:25   8.77G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:26   8.88G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:27   8.99G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:28   9.10G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:29   9.21G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:30   9.33G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:31   9.45G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:32   9.54G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:33   9.66G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:34   9.78G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:35   9.89G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:36   10.0G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:37   10.1G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:38   10.2G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:39   10.3G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:40   10.4G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:41   10.6G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:42   10.8G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:43   10.9G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:44   11.0G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:45   11.1G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:46   11.2G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:47   11.3G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:48   11.4G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:49   11.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:50   11.6G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:51   11.7G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:52   11.8G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:53   12.0G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:55   12.2G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:56   12.3G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:57   12.4G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:58   12.6G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:29:59   12.7G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:00   12.9G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:01   13.1G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:02   13.3G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:03   13.4G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:04   13.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:05   13.7G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:06   13.8G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:07   14.0G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:08   14.1G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:09   14.3G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:10   14.4G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:11   14.6G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:12   14.7G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:13   14.9G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:14   15.0G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:15   15.2G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:16   15.3G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:17   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:18   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:20   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:21   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:22   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:23   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:24   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:25   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:26   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:27   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:28   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:29   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:30   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:31   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:32   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:33   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:34   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:35   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:36   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:37   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:38   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:39   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:40   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:41   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:42   15.5G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:43   15.6G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:44   15.6G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
14:30:45   15.7G   projectzfspool/projectzfs/rhqbbar@day-17-20151217
Dec 22 14:30:47 (I) Creating snapshot nexsanpool1/rhqbbar@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/rhqbbar@day-18-20151218 estimated size is 3.60G
total estimated size is 3.60G
TIME        SENT   SNAPSHOT
14:30:49   3.56M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:50   9.84M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:51   12.2M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:52   22.4M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:53   35.6M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:54    165M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:55    284M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:56    412M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:57    536M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:58    657M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:30:59    782M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:00    919M   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:01   1.03G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:02   1.16G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:03   1.30G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:04   1.44G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:05   1.57G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:06   1.70G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:07   1.81G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:08   1.98G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:09   2.19G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:10   2.40G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:11   2.60G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:12   2.76G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:13   2.99G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:14   3.21G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:15   3.38G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:16   3.56G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:17   3.63G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:18   3.64G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
14:31:19   3.64G   projectzfspool/projectzfs/rhqbbar@day-18-20151218
Dec 22 14:31:27 (I) Creating snapshot nexsanpool1/rhqbbar@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/rhqbbar@day-19-20151219 estimated size is 18.7M
total estimated size is 18.7M
TIME        SENT   SNAPSHOT
14:31:28   5.84M   projectzfspool/projectzfs/rhqbbar@day-19-20151219
14:31:29   20.7M   projectzfspool/projectzfs/rhqbbar@day-19-20151219
Dec 22 14:31:31 (I) Creating snapshot nexsanpool1/rhqbbar@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/rhqbbar@day-20-20151220 estimated size is 6.90M
total estimated size is 6.90M
TIME        SENT   SNAPSHOT
14:31:32   5.75M   projectzfspool/projectzfs/rhqbbar@day-20-20151220
14:31:33   15.6M   projectzfspool/projectzfs/rhqbbar@day-20-20151220
Dec 22 14:31:34 (I) Creating snapshot nexsanpool1/rhqbbar@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/rhqbbar@day-21-20151221 estimated size is 14.3M
total estimated size is 14.3M
TIME        SENT   SNAPSHOT
14:31:35   7.33M   projectzfspool/projectzfs/rhqbbar@day-21-20151221
14:31:36   21.1M   projectzfspool/projectzfs/rhqbbar@day-21-20151221
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# cat /root/zfs-scripts/dataset-list-reduced | ./replica-zfs-snapshot.sh
Dec 22 14:32:21 (I) Snapshot date tag is 22

Dec 22 14:32:21 (I) Reading dataset list from stdin ..
Dec 22 14:32:21 (I) Replicating snapshots for axial
Dec 22 14:32:21 (I) Replicating snapshots for charmonium
Dec 22 14:32:21 (I) Replicating snapshots for c51
Dec 22 14:32:21 (I) Replicating snapshots for disco
Dec 22 14:32:21 (I) Replicating snapshots for fourpluseight
Dec 22 14:32:21 (I) Replicating snapshots for heavylight
Dec 22 14:32:21 (I) Creating snapshot nexsanpool1/heavylight@day-11-20151211 from incremental between projectzfspool/projectzfs/heavylight@day-16-20151216 and projectzfspool/projectzfs/heavylight@day-11-20151211
Dec 22 14:32:21 (I) Creating snapshot nexsanpool1/heavylight@day-13-20151213 from incremental between projectzfspool/projectzfs/heavylight@day-11-20151211 and projectzfspool/projectzfs/heavylight@day-13-20151213
Dec 22 14:32:21 (I) Creating snapshot nexsanpool1/heavylight@day-17-20151217 from incremental between projectzfspool/projectzfs/heavylight@day-13-20151213 and projectzfspool/projectzfs/heavylight@day-17-20151217
Dec 22 14:32:21 (I) Creating snapshot nexsanpool1/heavylight@day-18-20151218 from incremental between projectzfspool/projectzfs/heavylight@day-17-20151217 and projectzfspool/projectzfs/heavylight@day-18-20151218
Dec 22 14:32:21 (I) Creating snapshot nexsanpool1/heavylight@day-19-20151219 from incremental between projectzfspool/projectzfs/heavylight@day-18-20151218 and projectzfspool/projectzfs/heavylight@day-19-20151219
Dec 22 14:32:21 (I) Creating snapshot nexsanpool1/heavylight@day-20-20151220 from incremental between projectzfspool/projectzfs/heavylight@day-19-20151219 and projectzfspool/projectzfs/heavylight@day-20-20151220
Dec 22 14:32:21 (I) Creating snapshot nexsanpool1/heavylight@day-21-20151221 from incremental between projectzfspool/projectzfs/heavylight@day-20-20151220 and projectzfspool/projectzfs/heavylight@day-21-20151221
Dec 22 14:32:21 (I) Replicating snapshots for hpqcd
Dec 22 14:32:21 (I) Replicating snapshots for mixbk
Dec 22 14:32:21 (I) Replicating snapshots for nonperturbativestaggered
Dec 22 14:32:21 (I) Replicating snapshots for ok-action
Dec 22 14:32:21 (I) Replicating snapshots for rbcdwf
Dec 22 14:32:21 (I) Replicating snapshots for rhqbbar
[root@dslustre21 zfs.gba]# 

The last run I 'made it dry' to see why the heavylight backups failed. Looks like a backup was missing!

[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep heavylight
nexsanpool1/heavylight@day-1-20151201                                464K      -  2.76T  -
nexsanpool1/heavylight@day-2-20151202                                464K      -  2.76T  -
nexsanpool1/heavylight@day-3-20151203                               13.4M      -  2.76T  -
nexsanpool1/heavylight@day-4-20151204                               12.0M      -  2.76T  -
nexsanpool1/heavylight@day-5-20151205                               13.0M      -  2.76T  -
nexsanpool1/heavylight@day-6-20151206                               37.4M      -  2.76T  -
nexsanpool1/heavylight@day-7-20151207                               2.63G      -  2.77T  -
nexsanpool1/heavylight@day-8-20151208                                496K      -  2.77T  -
nexsanpool1/heavylight@day-9-20151209                                991K      -  2.77T  -
nexsanpool1/heavylight@day-10-20151210                               416K      -  2.77T  -
nexsanpool1/heavylight@day-12-20151212                               735K      -  2.77T  -
nexsanpool1/heavylight@day-14-20151214                              1.47M      -  2.77T  -
nexsanpool1/heavylight@day-15-20151215                              7.41M      -  2.77T  -
nexsanpool1/heavylight@day-16-20151216                                  0      -  2.77T  -
nexsanpool2/heavylight@day-1-20151201                                974M      -  2.76T  -
nexsanpool2/heavylight@week-1-20151207                              2.63G      -  2.77T  -
nexsanpool2/heavylight@week-2-20151214                                  0      -  2.77T  -
projectzfspool/projectzfs/heavylight@day-1-20151201                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-2-20151202                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-3-20151203                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-4-20151204                 6.75M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-5-20151205                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-6-20151206                 19.5M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-7-20151207                 2.50G      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-8-20151208                  264K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-9-20151209                  736K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-10-20151210                 192K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-11-20151211                9.05M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-12-20151212                 376K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-13-20151213                18.6M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-14-20151214                 752K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-15-20151215                4.60M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-16-20151216                3.29M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-17-20151217                4.30M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-18-20151218                6.13M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-19-20151219                16.6M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-20-20151220                6.68M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-21-20151221                1.56G      -  3.05T  -
[root@dslustre21 zfs.gba]# 

If I try to fix it manually I can't:

[root@dslustre21 zfs.gba]# zfs send -v -i projectzfspool/projectzfs/heavylight@day-10-20151210 projectzfspool/projectzfs/heavylight@day-11-20151211 | zfs recv nexsanpool1/heavylight@day-11-20151211
send from @day-10-20151210 to projectzfspool/projectzfs/heavylight@day-11-20151211 estimated size is 10.6M
total estimated size is 10.6M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: destination nexsanpool1/heavylight has been modified
since most recent snapshot
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-11-20151211': Broken pipe
[root@dslustre21 zfs.gba]# 

[root@dslustre21 zfs.gba]# zfs send -v -i projectzfspool/projectzfs/heavylight@day-12-20151212 projectzfspool/projectzfs/heavylight@day-13-20151213 | zfs recv nexsanpool1/heavylight@day-13-20151213
send from @day-12-20151212 to projectzfspool/projectzfs/heavylight@day-13-20151213 estimated size is 580M
total estimated size is 580M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: destination nexsanpool1/heavylight has been modified
since most recent snapshot
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-13-20151213': Broken pipe
[root@dslustre21 zfs.gba]# 

#8 Updated by Gerard Bernabeu Altayo almost 5 years ago

I can't make the 'heavylight' project snapshots to work... there is something wrong with those snapshots!

[root@dslustre21 zfs.gba]# zfs send -v -i projectzfspool/projectzfs/heavylight@day-10-20151210 projectzfspool/projectzfs/heavylight@day-11-20151211 | zfs recv nexsanpool1/heavylight@day-11-20151211
send from @day-10-20151210 to projectzfspool/projectzfs/heavylight@day-11-20151211 estimated size is 10.6M
total estimated size is 10.6M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: destination nexsanpool1/heavylight has been modified
since most recent snapshot
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-11-20151211': Broken pipe
[root@dslustre21 zfs.gba]# zfs destroy nexsanpool1/heavylight@day-16-20151216
[root@dslustre21 zfs.gba]# zfs destroy nexsanpool1/heavylight@day-15-20151215
[root@dslustre21 zfs.gba]# zfs destroy nexsanpool1/heavylight@day-14-20151214
[root@dslustre21 zfs.gba]# zfs destroy nexsanpool1/heavylight@day-13-20151213
could not find any snapshots to destroy; check snapshot names.
[root@dslustre21 zfs.gba]# zfs destroy nexsanpool1/heavylight@day-12-20151212
[root@dslustre21 zfs.gba]# cat /root/zfs-scripts/dataset-list-reduced | ./replica-zfs-snapshot.sh
Dec 22 14:41:17 (I) Snapshot date tag is 22

Dec 22 14:41:17 (I) Reading dataset list from stdin ..
Dec 22 14:41:17 (I) Replicating snapshots for axial
Dec 22 14:41:17 (I) Replicating snapshots for charmonium
Dec 22 14:41:17 (I) Replicating snapshots for c51
Dec 22 14:41:17 (I) Replicating snapshots for disco
Dec 22 14:41:17 (I) Replicating snapshots for fourpluseight
Dec 22 14:41:17 (I) Replicating snapshots for heavylight
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-11-20151211 from incremental between projectzfspool/projectzfs/heavylight@day-10-20151210 and projectzfspool/projectzfs/heavylight@day-11-20151211
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-12-20151212 from incremental between projectzfspool/projectzfs/heavylight@day-11-20151211 and projectzfspool/projectzfs/heavylight@day-12-20151212
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-13-20151213 from incremental between projectzfspool/projectzfs/heavylight@day-12-20151212 and projectzfspool/projectzfs/heavylight@day-13-20151213
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-14-20151214 from incremental between projectzfspool/projectzfs/heavylight@day-13-20151213 and projectzfspool/projectzfs/heavylight@day-14-20151214
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-15-20151215 from incremental between projectzfspool/projectzfs/heavylight@day-14-20151214 and projectzfspool/projectzfs/heavylight@day-15-20151215
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-16-20151216 from incremental between projectzfspool/projectzfs/heavylight@day-15-20151215 and projectzfspool/projectzfs/heavylight@day-16-20151216
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-17-20151217 from incremental between projectzfspool/projectzfs/heavylight@day-16-20151216 and projectzfspool/projectzfs/heavylight@day-17-20151217
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-18-20151218 from incremental between projectzfspool/projectzfs/heavylight@day-17-20151217 and projectzfspool/projectzfs/heavylight@day-18-20151218
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-19-20151219 from incremental between projectzfspool/projectzfs/heavylight@day-18-20151218 and projectzfspool/projectzfs/heavylight@day-19-20151219
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-20-20151220 from incremental between projectzfspool/projectzfs/heavylight@day-19-20151219 and projectzfspool/projectzfs/heavylight@day-20-20151220
Dec 22 14:41:18 (I) Creating snapshot nexsanpool1/heavylight@day-21-20151221 from incremental between projectzfspool/projectzfs/heavylight@day-20-20151220 and projectzfspool/projectzfs/heavylight@day-21-20151221
Dec 22 14:41:18 (I) Replicating snapshots for hpqcd
Dec 22 14:41:18 (I) Replicating snapshots for mixbk
Dec 22 14:41:18 (I) Replicating snapshots for nonperturbativestaggered
Dec 22 14:41:18 (I) Replicating snapshots for ok-action
Dec 22 14:41:18 (I) Replicating snapshots for rbcdwf
Dec 22 14:41:18 (I) Replicating snapshots for rhqbbar
[root@dslustre21 zfs.gba]# cat /root/zfs-scripts/dataset-list-reduced | ./replica-zfs-snapshot.sh
Dec 22 14:41:38 (I) Snapshot date tag is 22

Dec 22 14:41:38 (I) Reading dataset list from stdin ..
Dec 22 14:41:38 (I) Replicating snapshots for axial
Dec 22 14:41:38 (I) Replicating snapshots for charmonium
Dec 22 14:41:38 (I) Replicating snapshots for c51
Dec 22 14:41:38 (I) Replicating snapshots for disco
Dec 22 14:41:38 (I) Replicating snapshots for fourpluseight
Dec 22 14:41:38 (I) Replicating snapshots for heavylight
Dec 22 14:41:38 (I) Creating snapshot nexsanpool1/heavylight@day-11-20151211 from incremental between projectzfspool/projectzfs/heavylight@day-10-20151210 and projectzfspool/projectzfs/heavylight@day-11-20151211
send from @day-10-20151210 to projectzfspool/projectzfs/heavylight@day-11-20151211 estimated size is 10.6M
total estimated size is 10.6M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: destination nexsanpool1/heavylight has been modified
since most recent snapshot
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-11-20151211': Broken pipe
Dec 22 14:41:38 (I) Creating snapshot nexsanpool1/heavylight@day-12-20151212 from incremental between projectzfspool/projectzfs/heavylight@day-11-20151211 and projectzfspool/projectzfs/heavylight@day-12-20151212
send from @day-11-20151211 to projectzfspool/projectzfs/heavylight@day-12-20151212 estimated size is 259M
total estimated size is 259M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-12-20151212': Broken pipe
Dec 22 14:41:38 (I) Creating snapshot nexsanpool1/heavylight@day-13-20151213 from incremental between projectzfspool/projectzfs/heavylight@day-12-20151212 and projectzfspool/projectzfs/heavylight@day-13-20151213
send from @day-12-20151212 to projectzfspool/projectzfs/heavylight@day-13-20151213 estimated size is 580M
total estimated size is 580M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-13-20151213': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-14-20151214 from incremental between projectzfspool/projectzfs/heavylight@day-13-20151213 and projectzfspool/projectzfs/heavylight@day-14-20151214
send from @day-13-20151213 to projectzfspool/projectzfs/heavylight@day-14-20151214 estimated size is 654M
total estimated size is 654M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-14-20151214': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-15-20151215 from incremental between projectzfspool/projectzfs/heavylight@day-14-20151214 and projectzfspool/projectzfs/heavylight@day-15-20151215
send from @day-14-20151214 to projectzfspool/projectzfs/heavylight@day-15-20151215 estimated size is 1.75G
total estimated size is 1.75G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-15-20151215': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-16-20151216 from incremental between projectzfspool/projectzfs/heavylight@day-15-20151215 and projectzfspool/projectzfs/heavylight@day-16-20151216
send from @day-15-20151215 to projectzfspool/projectzfs/heavylight@day-16-20151216 estimated size is 2.64G
total estimated size is 2.64G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-16-20151216': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-17-20151217 from incremental between projectzfspool/projectzfs/heavylight@day-16-20151216 and projectzfspool/projectzfs/heavylight@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/heavylight@day-17-20151217 estimated size is 1.63G
total estimated size is 1.63G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-17-20151217': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-18-20151218 from incremental between projectzfspool/projectzfs/heavylight@day-17-20151217 and projectzfspool/projectzfs/heavylight@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/heavylight@day-18-20151218 estimated size is 5.86G
total estimated size is 5.86G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-18-20151218': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-19-20151219 from incremental between projectzfspool/projectzfs/heavylight@day-18-20151218 and projectzfspool/projectzfs/heavylight@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/heavylight@day-19-20151219 estimated size is 4.87G
total estimated size is 4.87G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-19-20151219': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-20-20151220 from incremental between projectzfspool/projectzfs/heavylight@day-19-20151219 and projectzfspool/projectzfs/heavylight@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/heavylight@day-20-20151220 estimated size is 4.89G
total estimated size is 4.89G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-20-20151220': Broken pipe
Dec 22 14:41:39 (I) Creating snapshot nexsanpool1/heavylight@day-21-20151221 from incremental between projectzfspool/projectzfs/heavylight@day-20-20151220 and projectzfspool/projectzfs/heavylight@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/heavylight@day-21-20151221 estimated size is 5.13G
total estimated size is 5.13G
TIME        SENT   SNAPSHOT
cannot receive incremental stream: most recent snapshot of nexsanpool1/heavylight does not
match incremental source
warning: cannot send 'projectzfspool/projectzfs/heavylight@day-21-20151221': Broken pipe
Dec 22 14:41:40 (I) Replicating snapshots for hpqcd
Dec 22 14:41:40 (I) Replicating snapshots for mixbk
Dec 22 14:41:40 (I) Replicating snapshots for nonperturbativestaggered
Dec 22 14:41:40 (I) Replicating snapshots for ok-action
Dec 22 14:41:40 (I) Replicating snapshots for rbcdwf
Dec 22 14:41:40 (I) Replicating snapshots for rhqbbar

#9 Updated by Gerard Bernabeu Altayo almost 5 years ago

Creating daily backups for everyone but heavylight, as I keep getting the replication error on them...

[root@dslustre21 ~]# cat /root/zfs-scripts/dataset-list-reduced | grep -v heavylight | /root/zfs.gba/replica-zfs-snapshot.sh

Second time it ran it ran clean (ie: succeeded to do all replications first):

[root@dslustre21 ~]# cat /root/zfs-scripts/dataset-list-reduced | grep -v heavylight | /root/zfs.gba/replica-zfs-snapshot.sh
Dec 24 06:27:50 (I) Snapshot date tag is 24

Dec 24 06:27:50 (I) Reading dataset list from stdin ..
Dec 24 06:27:50 (I) Replicating snapshots for axial
Dec 24 06:27:50 (I) Replicating snapshots for charmonium
Dec 24 06:27:50 (I) Replicating snapshots for c51
Dec 24 06:27:50 (I) Replicating snapshots for disco
Dec 24 06:27:50 (I) Replicating snapshots for fourpluseight
Dec 24 06:27:50 (I) Replicating snapshots for hpqcd
Dec 24 06:27:50 (I) Replicating snapshots for mixbk
Dec 24 06:27:50 (I) Replicating snapshots for nonperturbativestaggered
Dec 24 06:27:50 (I) Replicating snapshots for ok-action
Dec 24 06:27:50 (I) Replicating snapshots for rbcdwf
Dec 24 06:27:51 (I) Replicating snapshots for rhqbbar
[root@dslustre21 ~]# 

Will try to start replicating from heavylight and if I fail to do all snapshots I will wipe the copy from nexan1 and start it over (only for heavylight).

#10 Updated by Gerard Bernabeu Altayo almost 5 years ago

I've been reading lots of ZFS docs and the best route here did sound like using the -F option when receiving the zfs stream:

           -F

               Force  a  rollback  of the file system to the most recent snapshot before performing the receive operation. If receiving an incremental replication
               stream (for example, one generated by zfs send -R -[iI]), destroy snapshots and file systems that do not exist on the sending side.

Doing it seems to succeed:

[root@dslustre21 ~]# zfs send -v -i projectzfspool/projectzfs/heavylight@day-7-20151207 projectzfspool/projectzfs/heavylight@day-8-20151208 | zfs recv -F nexsanpool1/heavylight@day-8-20151208
send from @day-7-20151207 to projectzfspool/projectzfs/heavylight@day-8-20151208 estimated size is 4.92G
total estimated size is 4.92G
TIME        SENT   SNAPSHOT
06:32:57   15.2M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:32:58   15.2M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:32:59   48.1M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:00    169M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:01    290M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:02    420M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:03    547M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:04    683M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:05    795M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:06    916M   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:07   1.02G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:08   1.13G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:09   1.24G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:10   1.34G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:11   1.46G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:12   1.54G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:13   1.63G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:14   1.73G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:15   1.89G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:16   2.02G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:17   2.15G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:18   2.26G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:19   2.39G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:20   2.52G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:21   2.63G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:22   2.82G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:23   3.03G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:24   3.26G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:25   3.51G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:26   3.73G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:27   3.93G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:28   4.17G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:29   4.34G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:30   4.56G   projectzfspool/projectzfs/heavylight@day-8-20151208
06:33:31   4.76G   projectzfspool/projectzfs/heavylight@day-8-20151208
[root@dslustre21 ~]# man zfs
[root@dslustre21 ~]# zfs list -t snapshot | grep heavylight
nexsanpool1/heavylight@day-1-20151201                                464K      -  2.76T  -
nexsanpool1/heavylight@day-2-20151202                                464K      -  2.76T  -
nexsanpool1/heavylight@day-3-20151203                               13.4M      -  2.76T  -
nexsanpool1/heavylight@day-4-20151204                               12.0M      -  2.76T  -
nexsanpool1/heavylight@day-5-20151205                               13.0M      -  2.76T  -
nexsanpool1/heavylight@day-6-20151206                               37.4M      -  2.76T  -
nexsanpool1/heavylight@day-7-20151207                               2.63G      -  2.77T  -
nexsanpool1/heavylight@day-8-20151208                                   0      -  2.77T  -
nexsanpool2/heavylight@day-1-20151201                                974M      -  2.76T  -
nexsanpool2/heavylight@week-1-20151207                              2.63G      -  2.77T  -
nexsanpool2/heavylight@week-2-20151214                                  0      -  2.77T  -
projectzfspool/projectzfs/heavylight@day-1-20151201                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-2-20151202                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-3-20151203                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-4-20151204                 6.75M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-5-20151205                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-6-20151206                 19.5M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-7-20151207                 2.50G      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-8-20151208                  264K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-9-20151209                  736K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-10-20151210                 192K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-11-20151211                9.05M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-12-20151212                 376K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-13-20151213                18.6M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-14-20151214                 752K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-15-20151215                4.60M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-16-20151216                3.29M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-17-20151217                4.30M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-18-20151218                6.13M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-19-20151219                16.6M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-20-20151220                6.68M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-21-20151221                1.56G      -  3.05T  -
projectzfspool/projectzfs/heavylight@day-22-20151222                34.2M      -  3.05T  -
projectzfspool/projectzfs/heavylight@day-23-20151223                4.18M      -  3.05T  -
[root@dslustre21 ~]# 

Now the scipt works fine!

[root@dslustre21 ~]# zfs list -t snapshot | grep heavylight
nexsanpool1/heavylight@day-1-20151201                                464K      -  2.76T  -
nexsanpool1/heavylight@day-2-20151202                                464K      -  2.76T  -
nexsanpool1/heavylight@day-3-20151203                               13.4M      -  2.76T  -
nexsanpool1/heavylight@day-4-20151204                               12.0M      -  2.76T  -
nexsanpool1/heavylight@day-5-20151205                               13.0M      -  2.76T  -
nexsanpool1/heavylight@day-6-20151206                               37.4M      -  2.76T  -
nexsanpool1/heavylight@day-7-20151207                               2.63G      -  2.77T  -
nexsanpool1/heavylight@day-8-20151208                                   0      -  2.77T  -
nexsanpool2/heavylight@day-1-20151201                                974M      -  2.76T  -
nexsanpool2/heavylight@week-1-20151207                              2.63G      -  2.77T  -
nexsanpool2/heavylight@week-2-20151214                                  0      -  2.77T  -
projectzfspool/projectzfs/heavylight@day-1-20151201                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-2-20151202                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-3-20151203                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-4-20151204                 6.75M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-5-20151205                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-6-20151206                 19.5M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-7-20151207                 2.50G      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-8-20151208                  264K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-9-20151209                  736K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-10-20151210                 192K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-11-20151211                9.05M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-12-20151212                 376K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-13-20151213                18.6M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-14-20151214                 752K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-15-20151215                4.60M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-16-20151216                3.29M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-17-20151217                4.30M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-18-20151218                6.13M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-19-20151219                16.6M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-20-20151220                6.68M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-21-20151221                1.56G      -  3.05T  -
projectzfspool/projectzfs/heavylight@day-22-20151222                34.2M      -  3.05T  -
projectzfspool/projectzfs/heavylight@day-23-20151223                4.18M      -  3.05T  -
[root@dslustre21 ~]# cat /root/zfs-scripts/dataset-list-reduced | /root/zfs.gba/replica-zfs-snapshot.sh
Dec 24 06:35:21 (I) Snapshot date tag is 24

Dec 24 06:35:21 (I) Reading dataset list from stdin ..
Dec 24 06:35:21 (I) Replicating snapshots for axial
Dec 24 06:35:21 (I) Replicating snapshots for charmonium
Dec 24 06:35:21 (I) Replicating snapshots for c51
Dec 24 06:35:21 (I) Replicating snapshots for disco
Dec 24 06:35:21 (I) Replicating snapshots for fourpluseight
Dec 24 06:35:21 (I) Replicating snapshots for heavylight
Dec 24 06:35:21 (I) Creating snapshot nexsanpool1/heavylight@day-9-20151209 from incremental between projectzfspool/projectzfs/heavylight@day-8-20151208 and projectzfspool/projectzfs/heavylight@day-9-20151209
send from @day-8-20151208 to projectzfspool/projectzfs/heavylight@day-9-20151209 estimated size is 148M
total estimated size is 148M
TIME        SENT   SNAPSHOT
06:35:22   26.0M   projectzfspool/projectzfs/heavylight@day-9-20151209
Dec 24 06:35:24 (I) Creating snapshot nexsanpool1/heavylight@day-10-20151210 from incremental between projectzfspool/projectzfs/heavylight@day-9-20151209 and projectzfspool/projectzfs/heavylight@day-10-20151210
send from @day-9-20151209 to projectzfspool/projectzfs/heavylight@day-10-20151210 estimated size is 9.84M
total estimated size is 9.84M
TIME        SENT   SNAPSHOT
Dec 24 06:35:25 (I) Creating snapshot nexsanpool1/heavylight@day-11-20151211 from incremental between projectzfspool/projectzfs/heavylight@day-10-20151210 and projectzfspool/projectzfs/heavylight@day-11-20151211
send from @day-10-20151210 to projectzfspool/projectzfs/heavylight@day-11-20151211 estimated size is 10.6M
total estimated size is 10.6M
TIME        SENT   SNAPSHOT
Dec 24 06:35:26 (I) Creating snapshot nexsanpool1/heavylight@day-12-20151212 from incremental between projectzfspool/projectzfs/heavylight@day-11-20151211 and projectzfspool/projectzfs/heavylight@day-12-20151212
send from @day-11-20151211 to projectzfspool/projectzfs/heavylight@day-12-20151212 estimated size is 259M
total estimated size is 259M
TIME        SENT   SNAPSHOT
06:35:27   90.5M   projectzfspool/projectzfs/heavylight@day-12-20151212
06:35:28    255M   projectzfspool/projectzfs/heavylight@day-12-20151212
Dec 24 06:35:29 (I) Creating snapshot nexsanpool1/heavylight@day-13-20151213 from incremental between projectzfspool/projectzfs/heavylight@day-12-20151212 and projectzfspool/projectzfs/heavylight@day-13-20151213
send from @day-12-20151212 to projectzfspool/projectzfs/heavylight@day-13-20151213 estimated size is 580M
total estimated size is 580M
TIME        SENT   SNAPSHOT
06:35:30    104M   projectzfspool/projectzfs/heavylight@day-13-20151213
06:35:31    296M   projectzfspool/projectzfs/heavylight@day-13-20151213
06:35:32    439M   projectzfspool/projectzfs/heavylight@day-13-20151213
06:35:33    550M   projectzfspool/projectzfs/heavylight@day-13-20151213
Dec 24 06:35:34 (I) Creating snapshot nexsanpool1/heavylight@day-14-20151214 from incremental between projectzfspool/projectzfs/heavylight@day-13-20151213 and projectzfspool/projectzfs/heavylight@day-14-20151214
send from @day-13-20151213 to projectzfspool/projectzfs/heavylight@day-14-20151214 estimated size is 654M
total estimated size is 654M
TIME        SENT   SNAPSHOT
06:35:36   55.3M   projectzfspool/projectzfs/heavylight@day-14-20151214
06:35:37    147M   projectzfspool/projectzfs/heavylight@day-14-20151214
06:35:38    254M   projectzfspool/projectzfs/heavylight@day-14-20151214
06:35:39    394M   projectzfspool/projectzfs/heavylight@day-14-20151214
06:35:40    539M   projectzfspool/projectzfs/heavylight@day-14-20151214
Dec 24 06:35:43 (I) Creating snapshot nexsanpool1/heavylight@day-15-20151215 from incremental between projectzfspool/projectzfs/heavylight@day-14-20151214 and projectzfspool/projectzfs/heavylight@day-15-20151215
send from @day-14-20151214 to projectzfspool/projectzfs/heavylight@day-15-20151215 estimated size is 1.75G
total estimated size is 1.75G
TIME        SENT   SNAPSHOT
06:35:44   28.2M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:45    120M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:46    217M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:47    306M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:48    424M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:49    545M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:50    641M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:51    697M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:52    755M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:53    864M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:54    978M   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:55   1.10G   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:56   1.22G   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:57   1.38G   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:58   1.54G   projectzfspool/projectzfs/heavylight@day-15-20151215
06:35:59   1.64G   projectzfspool/projectzfs/heavylight@day-15-20151215
06:36:00   1.71G   projectzfspool/projectzfs/heavylight@day-15-20151215
Dec 24 06:36:01 (I) Creating snapshot nexsanpool1/heavylight@day-16-20151216 from incremental between projectzfspool/projectzfs/heavylight@day-15-20151215 and projectzfspool/projectzfs/heavylight@day-16-20151216
send from @day-15-20151215 to projectzfspool/projectzfs/heavylight@day-16-20151216 estimated size is 2.64G
total estimated size is 2.64G
TIME        SENT   SNAPSHOT
06:36:02   4.50M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:03   76.9M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:04    177M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:05    297M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:06    431M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:07    564M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:08    678M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:09    774M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:10    854M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:11    928M   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:12   1.03G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:13   1.17G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:14   1.34G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:15   1.51G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:16   1.64G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:17   1.75G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:18   1.83G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:19   1.95G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:20   2.04G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:21   2.18G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:22   2.35G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:23   2.51G   projectzfspool/projectzfs/heavylight@day-16-20151216
06:36:24   2.61G   projectzfspool/projectzfs/heavylight@day-16-20151216
Dec 24 06:36:26 (I) Creating snapshot nexsanpool1/heavylight@day-17-20151217 from incremental between projectzfspool/projectzfs/heavylight@day-16-20151216 and projectzfspool/projectzfs/heavylight@day-17-20151217
send from @day-16-20151216 to projectzfspool/projectzfs/heavylight@day-17-20151217 estimated size is 1.63G
total estimated size is 1.63G
TIME        SENT   SNAPSHOT
06:36:27   17.1M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:28    135M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:29    266M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:30    386M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:31    532M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:32    714M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:33    857M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:34    983M   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:35   1.03G   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:36   1.13G   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:37   1.25G   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:38   1.41G   projectzfspool/projectzfs/heavylight@day-17-20151217
06:36:39   1.54G   projectzfspool/projectzfs/heavylight@day-17-20151217
Dec 24 06:36:41 (I) Creating snapshot nexsanpool1/heavylight@day-18-20151218 from incremental between projectzfspool/projectzfs/heavylight@day-17-20151217 and projectzfspool/projectzfs/heavylight@day-18-20151218
send from @day-17-20151217 to projectzfspool/projectzfs/heavylight@day-18-20151218 estimated size is 5.86G
total estimated size is 5.86G
TIME        SENT   SNAPSHOT
06:36:42   60.5M   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:43    249M   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:44    398M   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:45    585M   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:46    775M   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:47    933M   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:48   1.10G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:49   1.24G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:50   1.42G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:51   1.56G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:52   1.75G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:53   1.88G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:54   2.04G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:55   2.18G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:56   2.36G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:57   2.54G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:58   2.71G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:36:59   2.87G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:00   3.05G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:01   3.20G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:02   3.42G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:03   3.58G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:04   3.77G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:05   3.95G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:06   4.12G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:07   4.32G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:08   4.49G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:09   4.67G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:10   4.84G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:11   4.91G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:12   5.00G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:13   5.02G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:14   5.04G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:15   5.08G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:16   5.11G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:17   5.17G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:18   5.25G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:19   5.35G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:20   5.44G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:21   5.51G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:22   5.56G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:23   5.64G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:24   5.76G   projectzfspool/projectzfs/heavylight@day-18-20151218
06:37:25   5.86G   projectzfspool/projectzfs/heavylight@day-18-20151218
Dec 24 06:37:28 (I) Creating snapshot nexsanpool1/heavylight@day-19-20151219 from incremental between projectzfspool/projectzfs/heavylight@day-18-20151218 and projectzfspool/projectzfs/heavylight@day-19-20151219
send from @day-18-20151218 to projectzfspool/projectzfs/heavylight@day-19-20151219 estimated size is 4.87G
total estimated size is 4.87G
TIME        SENT   SNAPSHOT
06:37:29   8.04M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:30   11.6M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:31   14.8M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:32   41.2M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:33    143M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:34    225M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:35    324M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:36    417M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:37    499M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:38    590M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:39    671M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:40    754M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:41    832M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:42    920M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:43   1005M   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:44   1.06G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:45   1.14G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:46   1.22G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:47   1.30G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:48   1.36G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:49   1.44G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:50   1.53G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:51   1.61G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:52   1.71G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:53   1.78G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:54   1.87G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:55   1.94G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:56   2.03G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:57   2.12G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:58   2.20G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:37:59   2.28G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:00   2.35G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:01   2.46G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:02   2.58G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:03   2.70G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:04   2.80G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:05   2.90G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:06   2.96G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:07   3.02G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:08   3.08G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:09   3.14G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:10   3.24G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:11   3.32G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:12   3.42G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:13   3.49G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:14   3.60G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:15   3.71G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:16   3.84G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:17   3.93G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:18   3.98G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:19   4.03G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:20   4.11G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:21   4.20G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:22   4.30G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:23   4.39G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:24   4.48G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:25   4.58G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:26   4.68G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:27   4.76G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:28   4.85G   projectzfspool/projectzfs/heavylight@day-19-20151219
06:38:29   4.89G   projectzfspool/projectzfs/heavylight@day-19-20151219
Dec 24 06:38:30 (I) Creating snapshot nexsanpool1/heavylight@day-20-20151220 from incremental between projectzfspool/projectzfs/heavylight@day-19-20151219 and projectzfspool/projectzfs/heavylight@day-20-20151220
send from @day-19-20151219 to projectzfspool/projectzfs/heavylight@day-20-20151220 estimated size is 4.89G
total estimated size is 4.89G
TIME        SENT   SNAPSHOT
06:38:31   4.38M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:32   10.5M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:33   14.7M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:34   22.8M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:35   73.8M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:36    161M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:37    230M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:38    318M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:39    403M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:40    487M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:41    576M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:42    659M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:43    742M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:44    819M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:45    909M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:46    992M   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:47   1.04G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:48   1.12G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:49   1.19G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:50   1.29G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:51   1.38G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:52   1.47G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:53   1.56G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:54   1.65G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:55   1.73G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:56   1.81G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:57   1.90G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:58   1.98G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:38:59   2.06G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:00   2.15G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:01   2.18G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:02   2.21G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:03   2.28G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:04   2.36G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:05   2.42G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:06   2.48G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:07   2.56G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:08   2.64G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:09   2.72G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:10   2.80G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:11   2.90G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:12   2.96G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:13   3.03G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:14   3.09G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:15   3.14G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:16   3.20G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:17   3.25G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:18   3.31G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:19   3.37G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:20   3.44G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:21   3.49G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:22   3.56G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:23   3.61G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:24   3.68G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:25   3.73G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:26   3.79G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:27   3.86G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:28   3.92G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:29   3.96G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:30   4.01G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:31   4.05G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:32   4.11G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:33   4.17G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:34   4.25G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:35   4.33G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:36   4.40G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:37   4.48G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:38   4.54G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:39   4.63G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:40   4.72G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:41   4.81G   projectzfspool/projectzfs/heavylight@day-20-20151220
06:39:42   4.90G   projectzfspool/projectzfs/heavylight@day-20-20151220
Dec 24 06:39:43 (I) Creating snapshot nexsanpool1/heavylight@day-21-20151221 from incremental between projectzfspool/projectzfs/heavylight@day-20-20151220 and projectzfspool/projectzfs/heavylight@day-21-20151221
send from @day-20-20151220 to projectzfspool/projectzfs/heavylight@day-21-20151221 estimated size is 5.13G
total estimated size is 5.13G
TIME        SENT   SNAPSHOT
06:39:44    501K   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:45   18.3M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:46    114M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:47    193M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:48    273M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:49    360M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:50    455M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:51    532M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:52    611M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:53    690M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:54    782M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:55    869M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:56    940M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:57   1020M   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:58   1.08G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:39:59   1.17G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:00   1.25G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:01   1.33G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:02   1.41G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:03   1.49G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:04   1.58G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:05   1.67G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:06   1.76G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:07   1.85G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:08   1.93G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:09   2.03G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:10   2.12G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:11   2.22G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:12   2.27G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:13   2.32G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:14   2.38G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:15   2.50G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:16   2.63G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:17   2.75G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:18   2.83G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:19   2.91G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:20   3.00G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:21   3.08G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:22   3.16G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:23   3.21G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:24   3.27G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:25   3.31G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:26   3.38G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:27   3.45G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:28   3.53G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:29   3.60G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:30   3.65G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:31   3.72G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:32   3.80G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:33   3.88G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:34   3.96G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:35   4.02G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:36   4.12G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:37   4.22G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:38   4.32G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:39   4.39G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:40   4.49G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:41   4.61G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:42   4.76G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:43   4.87G   projectzfspool/projectzfs/heavylight@day-21-20151221
06:40:44   5.04G   projectzfspool/projectzfs/heavylight@day-21-20151221
Dec 24 06:40:47 (I) Creating snapshot nexsanpool1/heavylight@day-22-20151222 from incremental between projectzfspool/projectzfs/heavylight@day-21-20151221 and projectzfspool/projectzfs/heavylight@day-22-20151222
send from @day-21-20151221 to projectzfspool/projectzfs/heavylight@day-22-20151222 estimated size is 8.52G
total estimated size is 8.52G
TIME        SENT   SNAPSHOT
06:40:48   7.54M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:49   7.54M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:50   7.54M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:51   8.29M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:52   9.32M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:53   10.1M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:54   11.8M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:55   12.8M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:56   14.6M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:57   15.9M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:58   59.1M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:40:59    193M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:00    383M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:01    568M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:02    760M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:03    962M   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:04   1.13G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:05   1.33G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:06   1.53G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:07   1.72G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:08   1.90G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:09   2.06G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:10   2.26G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:11   2.45G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:12   2.63G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:13   2.81G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:14   2.98G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:15   3.17G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:16   3.35G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:17   3.55G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:18   3.73G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:19   3.91G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:20   4.10G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:21   4.35G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:22   4.59G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:23   4.84G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:24   5.07G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:25   5.31G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:26   5.54G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:27   5.77G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:28   6.01G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:29   6.23G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:30   6.40G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:31   6.60G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:32   6.83G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:33   7.03G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:34   7.24G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:35   7.46G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:36   7.68G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:37   7.91G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:38   8.13G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:39   8.35G   projectzfspool/projectzfs/heavylight@day-22-20151222
06:41:40   8.53G   projectzfspool/projectzfs/heavylight@day-22-20151222
Dec 24 06:41:48 (I) Creating snapshot nexsanpool1/heavylight@day-23-20151223 from incremental between projectzfspool/projectzfs/heavylight@day-22-20151222 and projectzfspool/projectzfs/heavylight@day-23-20151223
send from @day-22-20151222 to projectzfspool/projectzfs/heavylight@day-23-20151223 estimated size is 73.2M
total estimated size is 73.2M
TIME        SENT   SNAPSHOT
06:41:49   11.0M   projectzfspool/projectzfs/heavylight@day-23-20151223
06:41:50   15.7M   projectzfspool/projectzfs/heavylight@day-23-20151223
06:41:51   20.5M   projectzfspool/projectzfs/heavylight@day-23-20151223
06:41:52   31.1M   projectzfspool/projectzfs/heavylight@day-23-20151223
06:41:53   49.3M   projectzfspool/projectzfs/heavylight@day-23-20151223
06:41:54   59.0M   projectzfspool/projectzfs/heavylight@day-23-20151223
06:41:55   64.1M   projectzfspool/projectzfs/heavylight@day-23-20151223
06:41:56    126M   projectzfspool/projectzfs/heavylight@day-23-20151223
Dec 24 06:41:58 (I) Replicating snapshots for hpqcd
Dec 24 06:41:58 (I) Replicating snapshots for mixbk
Dec 24 06:41:58 (I) Replicating snapshots for nonperturbativestaggered
Dec 24 06:41:58 (I) Replicating snapshots for ok-action
Dec 24 06:41:58 (I) Replicating snapshots for rbcdwf
Dec 24 06:41:58 (I) Replicating snapshots for rhqbbar
[root@dslustre21 ~]# cat /root/zfs-scripts/dataset-list-reduced | /root/zfs.gba/replica-zfs-snapshot.sh
Dec 24 06:42:17 (I) Snapshot date tag is 24

Dec 24 06:42:17 (I) Reading dataset list from stdin ..
Dec 24 06:42:17 (I) Replicating snapshots for axial
Dec 24 06:42:18 (I) Replicating snapshots for charmonium
Dec 24 06:42:18 (I) Replicating snapshots for c51
Dec 24 06:42:18 (I) Replicating snapshots for disco
Dec 24 06:42:18 (I) Replicating snapshots for fourpluseight
Dec 24 06:42:18 (I) Replicating snapshots for heavylight
Dec 24 06:42:18 (I) Replicating snapshots for hpqcd
Dec 24 06:42:18 (I) Replicating snapshots for mixbk
Dec 24 06:42:18 (I) Replicating snapshots for nonperturbativestaggered
Dec 24 06:42:18 (I) Replicating snapshots for ok-action
Dec 24 06:42:18 (I) Replicating snapshots for rbcdwf
Dec 24 06:42:18 (I) Replicating snapshots for rhqbbar
[root@dslustre21 ~]# zfs list -t snapshot | grep heavylight
nexsanpool1/heavylight@day-1-20151201                                464K      -  2.76T  -
nexsanpool1/heavylight@day-2-20151202                                464K      -  2.76T  -
nexsanpool1/heavylight@day-3-20151203                               13.4M      -  2.76T  -
nexsanpool1/heavylight@day-4-20151204                               12.0M      -  2.76T  -
nexsanpool1/heavylight@day-5-20151205                               13.0M      -  2.76T  -
nexsanpool1/heavylight@day-6-20151206                               37.4M      -  2.76T  -
nexsanpool1/heavylight@day-7-20151207                               2.63G      -  2.77T  -
nexsanpool1/heavylight@day-8-20151208                                496K      -  2.77T  -
nexsanpool1/heavylight@day-9-20151209                                991K      -  2.77T  -
nexsanpool1/heavylight@day-10-20151210                               352K      -  2.77T  -
nexsanpool1/heavylight@day-11-20151211                              13.0M      -  2.77T  -
nexsanpool1/heavylight@day-12-20151212                               687K      -  2.77T  -
nexsanpool1/heavylight@day-13-20151213                              19.3M      -  2.77T  -
nexsanpool1/heavylight@day-14-20151214                              1.44M      -  2.77T  -
nexsanpool1/heavylight@day-15-20151215                              7.39M      -  2.77T  -
nexsanpool1/heavylight@day-16-20151216                              6.28M      -  2.77T  -
nexsanpool1/heavylight@day-17-20151217                              8.57M      -  2.77T  -
nexsanpool1/heavylight@day-18-20151218                              11.5M      -  2.78T  -
nexsanpool1/heavylight@day-19-20151219                              30.2M      -  2.78T  -
nexsanpool1/heavylight@day-20-20151220                              12.2M      -  2.78T  -
nexsanpool1/heavylight@day-21-20151221                              1.65G      -  2.79T  -
nexsanpool1/heavylight@day-22-20151222                              50.4M      -  2.79T  -
nexsanpool1/heavylight@day-23-20151223                                  0      -  2.79T  -
nexsanpool2/heavylight@day-1-20151201                                974M      -  2.76T  -
nexsanpool2/heavylight@week-1-20151207                              2.63G      -  2.77T  -
nexsanpool2/heavylight@week-2-20151214                                  0      -  2.77T  -
projectzfspool/projectzfs/heavylight@day-1-20151201                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-2-20151202                  232K      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-3-20151203                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-4-20151204                 6.75M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-5-20151205                 7.31M      -  3.02T  -
projectzfspool/projectzfs/heavylight@day-6-20151206                 19.5M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-7-20151207                 2.50G      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-8-20151208                  264K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-9-20151209                  736K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-10-20151210                 192K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-11-20151211                9.05M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-12-20151212                 376K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-13-20151213                18.6M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-14-20151214                 752K      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-15-20151215                4.60M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-16-20151216                3.29M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-17-20151217                4.30M      -  3.03T  -
projectzfspool/projectzfs/heavylight@day-18-20151218                6.13M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-19-20151219                16.6M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-20-20151220                6.68M      -  3.04T  -
projectzfspool/projectzfs/heavylight@day-21-20151221                1.56G      -  3.05T  -
projectzfspool/projectzfs/heavylight@day-22-20151222                34.2M      -  3.05T  -
projectzfspool/projectzfs/heavylight@day-23-20151223                4.18M      -  3.05T  -
[root@dslustre21 ~]# zpool list
NAME             SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
nexsanpool1     16.2T  7.70T  8.55T    47%  1.00x  ONLINE  -
nexsanpool2     16.2T  12.9T  3.37T    79%  1.00x  ONLINE  -
projectzfspool  7.25T  5.58T  1.67T    77%  1.00x  ONLINE  -
[root@dslustre21 ~]# 

All daily backups are done now :)

#11 Updated by Gerard Bernabeu Altayo almost 5 years ago

For the weeklys, I have the logic working, need to create the snapshots now:

[root@dslustre21 zfs.gba]# echo charmonium | ./weekly-zfs-snapshot.sh 
Dec 24 09:44:42 (I) Making weekly snapshots - Snapshot date tag is 24
Dec 24 09:44:42 (I) Reading dataset list from stdin ..
Dec 24 09:44:42 (I) Making snapshots for charmonium
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create
 week-3-20151221
[root@dslustre21 zfs.gba]# 

I'm following the same principle that the script can be executed as often as desired, this way I can merge everything in the daily script later on :)

#12 Updated by Gerard Bernabeu Altayo almost 5 years ago

My script to generate the weekly updates (once day-1 exists on nexsan2) works:

[root@dslustre21 zfs.gba]# echo charmonium | ./weekly-zfs-snapshot.sh 
Dec 24 13:04:45 (I) Making weekly snapshots - Snapshot date tag is 24
Dec 24 13:04:45 (I) Reading dataset list from stdin ..
Dec 24 13:04:45 (I) Making snapshots for charmonium
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create
 week-3-20151221
3
 14
 21
Dec 24 13:04:46 (I) zfs send -v -i nexsanpool1/charmonium@day-14-20151214 nexsanpool1/charmonium@day-21-20151221 | zfs recv nexsanpool2/charmonium@week-3-20151221
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep charmonium
nexsanpool1/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool1/charmonium@day-2-20151202                               16.0K      -   272K  -
nexsanpool1/charmonium@day-3-20151203                               16.0K      -   272K  -
nexsanpool1/charmonium@day-4-20151204                               16.0K      -   272K  -
nexsanpool1/charmonium@day-5-20151205                               16.0K      -   272K  -
nexsanpool1/charmonium@day-6-20151206                               16.0K      -   272K  -
nexsanpool1/charmonium@day-7-20151207                               16.0K      -   272K  -
nexsanpool1/charmonium@day-8-20151208                               16.0K      -   272K  -
nexsanpool1/charmonium@day-9-20151209                               16.0K      -   272K  -
nexsanpool1/charmonium@day-10-20151210                              16.0K      -   272K  -
nexsanpool1/charmonium@day-11-20151211                              16.0K      -   272K  -
nexsanpool1/charmonium@day-12-20151212                              16.0K      -   272K  -
nexsanpool1/charmonium@day-13-20151213                              16.0K      -   272K  -
nexsanpool1/charmonium@day-14-20151214                              16.0K      -   272K  -
nexsanpool1/charmonium@day-15-20151215                              16.0K      -   272K  -
nexsanpool1/charmonium@day-16-20151216                              16.0K      -   272K  -
nexsanpool1/charmonium@day-17-20151217                              16.0K      -   272K  -
nexsanpool1/charmonium@day-18-20151218                              16.0K      -   272K  -
nexsanpool1/charmonium@day-19-20151219                              16.0K      -   272K  -
nexsanpool1/charmonium@day-20-20151220                              16.0K      -   272K  -
nexsanpool1/charmonium@day-21-20151221                              16.0K      -   272K  -
nexsanpool1/charmonium@day-22-20151222                              16.0K      -   272K  -
nexsanpool1/charmonium@day-23-20151223                                  0      -   272K  -
nexsanpool2/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool2/charmonium@week-1-20151207                              16.0K      -   272K  -
nexsanpool2/charmonium@week-2-20151214                                  0      -   272K  -
projectzfspool/projectzfs/charmonium@day-1-20151201                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20151202                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20151203                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20151204                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-5-20151205                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-6-20151206                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-7-20151207                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-8-20151208                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-9-20151209                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-10-20151210                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-11-20151211                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-12-20151212                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-13-20151213                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-14-20151214                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-15-20151215                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-16-20151216                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-17-20151217                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-18-20151218                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-19-20151219                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-20-20151220                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-21-20151221                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-22-20151222                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-23-20151223                    0      -   136K  -
[root@dslustre21 zfs.gba]# vim weekly-zfs-snapshot.sh 
[root@dslustre21 zfs.gba]# zfs send -v -i nexsanpool1/charmonium@day-14-20151214 nexsanpool1/charmonium@day-21-20151221 | zfs recv nexsanpool2/charmonium@week-3-20151221
send from @day-14-20151214 to nexsanpool1/charmonium@day-21-20151221 estimated size is 512
total estimated size is 512
TIME        SENT   SNAPSHOT
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep charmonium
nexsanpool1/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool1/charmonium@day-2-20151202                               16.0K      -   272K  -
nexsanpool1/charmonium@day-3-20151203                               16.0K      -   272K  -
nexsanpool1/charmonium@day-4-20151204                               16.0K      -   272K  -
nexsanpool1/charmonium@day-5-20151205                               16.0K      -   272K  -
nexsanpool1/charmonium@day-6-20151206                               16.0K      -   272K  -
nexsanpool1/charmonium@day-7-20151207                               16.0K      -   272K  -
nexsanpool1/charmonium@day-8-20151208                               16.0K      -   272K  -
nexsanpool1/charmonium@day-9-20151209                               16.0K      -   272K  -
nexsanpool1/charmonium@day-10-20151210                              16.0K      -   272K  -
nexsanpool1/charmonium@day-11-20151211                              16.0K      -   272K  -
nexsanpool1/charmonium@day-12-20151212                              16.0K      -   272K  -
nexsanpool1/charmonium@day-13-20151213                              16.0K      -   272K  -
nexsanpool1/charmonium@day-14-20151214                              16.0K      -   272K  -
nexsanpool1/charmonium@day-15-20151215                              16.0K      -   272K  -
nexsanpool1/charmonium@day-16-20151216                              16.0K      -   272K  -
nexsanpool1/charmonium@day-17-20151217                              16.0K      -   272K  -
nexsanpool1/charmonium@day-18-20151218                              16.0K      -   272K  -
nexsanpool1/charmonium@day-19-20151219                              16.0K      -   272K  -
nexsanpool1/charmonium@day-20-20151220                              16.0K      -   272K  -
nexsanpool1/charmonium@day-21-20151221                              16.0K      -   272K  -
nexsanpool1/charmonium@day-22-20151222                              16.0K      -   272K  -
nexsanpool1/charmonium@day-23-20151223                                  0      -   272K  -
nexsanpool2/charmonium@day-1-20151201                               16.0K      -   272K  -
nexsanpool2/charmonium@week-1-20151207                              16.0K      -   272K  -
nexsanpool2/charmonium@week-2-20151214                              16.0K      -   272K  -
nexsanpool2/charmonium@week-3-20151221                                  0      -   272K  -
projectzfspool/projectzfs/charmonium@day-1-20151201                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20151202                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20151203                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20151204                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-5-20151205                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-6-20151206                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-7-20151207                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-8-20151208                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-9-20151209                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-10-20151210                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-11-20151211                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-12-20151212                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-13-20151213                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-14-20151214                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-15-20151215                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-16-20151216                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-17-20151217                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-18-20151218                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-19-20151219                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-20-20151220                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-21-20151221                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-22-20151222                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-23-20151223                    0      -   136K  -
[root@dslustre21 zfs.gba]# echo charmonium | ./weekly-zfs-snapshot.sh 
Dec 24 13:06:44 (I) Making weekly snapshots - Snapshot date tag is 24
Dec 24 13:06:44 (I) Reading dataset list from stdin ..
Dec 24 13:06:44 (I) Making snapshots for charmonium
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create

[root@dslustre21 zfs.gba]# 

Going to finish some stuff to leave it running (needs to move some serious data for some).

#13 Updated by Gerard Bernabeu Altayo almost 5 years ago

running (interactively):

[root@dslustre21 zfs.gba]# echo rhqbbar | ./weekly-zfs-snapshot.sh 
Dec 24 13:11:49 (I) Making weekly snapshots - Snapshot date tag is 24
Dec 24 13:11:49 (I) Reading dataset list from stdin ..
Dec 24 13:11:49 (I) Making snapshots for rhqbbar
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create
 week-3-20151221
Dec 24 13:11:49 (I) zfs send -v -i nexsanpool1/rhqbbar@day-14-20151214 nexsanpool1/rhqbbar@day-21-20151221 | zfs recv nexsanpool2/rhqbbar@week-3-20151221
[root@dslustre21 zfs.gba]# vim weekly-zfs-snapshot.sh 
[root@dslustre21 zfs.gba]# cat /root/zfs-scripts/dataset-list-reduced | /root/zfs.gba/weekly-zfs-snapshot.sh 
Dec 24 13:13:19 (I) Making weekly snapshots - Snapshot date tag is 24
Dec 24 13:13:19 (I) Reading dataset list from stdin ..
Dec 24 13:13:19 (I) Making snapshots for axial
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create
 week-3-20151221
Dec 24 13:13:19 (I) zfs send -v -i nexsanpool1/axial@day-14-20151214 nexsanpool1/axial@day-21-20151221 | zfs recv nexsanpool2/axial@week-3-20151221
send from @day-14-20151214 to nexsanpool1/axial@day-21-20151221 estimated size is 512
total estimated size is 512
TIME        SENT   SNAPSHOT
Dec 24 13:13:20 (I) Making snapshots for charmonium
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create

Dec 24 13:13:20 (I) Making snapshots for c51
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create
 week-3-20151221
Dec 24 13:13:20 (I) zfs send -v -i nexsanpool1/c51@day-14-20151214 nexsanpool1/c51@day-21-20151221 | zfs recv nexsanpool2/c51@week-3-20151221
send from @day-14-20151214 to nexsanpool1/c51@day-21-20151221 estimated size is 512
total estimated size is 512
TIME        SENT   SNAPSHOT
Dec 24 13:13:20 (I) Making snapshots for disco
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create
 week-3-20151221
Dec 24 13:13:20 (I) zfs send -v -i nexsanpool1/disco@day-14-20151214 nexsanpool1/disco@day-21-20151221 | zfs recv nexsanpool2/disco@week-3-20151221
send from @day-14-20151214 to nexsanpool1/disco@day-21-20151221 estimated size is 512
total estimated size is 512
TIME        SENT   SNAPSHOT
Dec 24 13:13:21 (I) Making snapshots for fourpluseight
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221
Will create
 week-3-20151221
Dec 24 13:13:21 (I) zfs send -v -i nexsanpool1/fourpluseight@day-14-20151214 nexsanpool1/fourpluseight@day-21-20151221 | zfs recv nexsanpool2/fourpluseight@week-3-20151221
send from @day-14-20151214 to nexsanpool1/fourpluseight@day-21-20151221 estimated size is 1.42G
total estimated size is 1.42G
TIME        SENT   SNAPSHOT
13:13:22   39.0M   nexsanpool1/fourpluseight@day-21-20151221
13:13:23   69.3M   nexsanpool1/fourpluseight@day-21-20151221
13:13:24    103M   nexsanpool1/fourpluseight@day-21-20151221
13:13:25    143M   nexsanpool1/fourpluseight@day-21-20151221
13:13:26    169M   nexsanpool1/fourpluseight@day-21-20151221
13:13:27    198M   nexsanpool1/fourpluseight@day-21-20151221
13:13:28    227M   nexsanpool1/fourpluseight@day-21-20151221
13:13:29    257M   nexsanpool1/fourpluseight@day-21-20151221
13:13:30    277M   nexsanpool1/fourpluseight@day-21-20151221
13:13:31    281M   nexsanpool1/fourpluseight@day-21-20151221
13:13:32    292M   nexsanpool1/fourpluseight@day-21-20151221
13:13:33    342M   nexsanpool1/fourpluseight@day-21-20151221
13:13:34    447M   nexsanpool1/fourpluseight@day-21-20151221
13:13:35    454M   nexsanpool1/fourpluseight@day-21-20151221
13:13:36    459M   nexsanpool1/fourpluseight@day-21-20151221
13:13:37    467M   nexsanpool1/fourpluseight@day-21-20151221
13:13:38    472M   nexsanpool1/fourpluseight@day-21-20151221
13:13:39    482M   nexsanpool1/fourpluseight@day-21-20151221
...

Right now it won't create day1 if it does not exist, will try and fail to do some zfs send/receive with invalid arguments...

#14 Updated by Gerard Bernabeu Altayo almost 5 years ago

it did work fine, now working on the GZIPs on weekly-zfs-gzip.sh

#15 Updated by Gerard Bernabeu Altayo almost 5 years ago

The script to make the GZIPs is ready, testing shows it would create the right zips:

[root@dslustre21 zfs.gba]# echo rhqbbar | ./weekly-zfs-gzip.sh 
Dec 28 11:32:20 (I) Making day1 and weekly snapshot GZIPs
Dec 28 11:32:20 (I) Reading dataset list from stdin ..
Dec 28 11:32:20 (I) Making GZIP for rhqbbar
Dec 28 11:32:20 (I) Creating /nexsanpool2/rhqbbar-day-1-20151201.gz with: zfs send nexsanpool2/rhqbbar@day-1-20151201 | gzip > /nexsanpool2/rhqbbar-day-1-20151201.gz &
Dec 28 11:32:20 (I) Creating /nexsanpool2/rhqbbar-week-1-20151207.gz with: zfs send -i nexsanpool2/rhqbbar@day-1-20151201 nexsanpool2/rhqbbar@week-1-20151207 | gzip > /nexsanpool2/rhqbbar-week-1-20151207.gz &
Dec 28 11:32:20 (I) Creating /nexsanpool2/rhqbbar-week-2-20151214.gz with: zfs send -i nexsanpool2/rhqbbar@week-1-20151207 nexsanpool2/rhqbbar@week-2-20151214 | gzip > /nexsanpool2/rhqbbar-week-2-20151214.gz &
Dec 28 11:32:20 (I) Creating /nexsanpool2/rhqbbar-week-3-20151221.gz with: zfs send -i nexsanpool2/rhqbbar@week-2-20151214 nexsanpool2/rhqbbar@week-3-20151221 | gzip > /nexsanpool2/rhqbbar-week-3-20151221.gz &
[root@dslustre21 zfs.gba]# 

With the current status, only a few ZIPs will be created:

[root@dslustre21 zfs.gba]# date; ll /nexsanpool2/rhqbbar*; nohup echo rhqbbar | ./weekly-zfs-gzip.sh & cat nohup.out
Mon Dec 28 11:39:04 CST 2015
-rw-r--r--  1 root  root    704609531459 Dec 11 08:27 /nexsanpool2/rhqbbar-day-1-20151201.gz
-rw-r--r--  1 root  root             148 Dec 10 14:15 /nexsanpool2/rhqbbar-week-1-20151207.gz

[1] 14399
cat: nohup.outnohup: ignoring input and redirecting stderr to stdout
: No such file or directory
[root@dslustre21 zfs.gba]# Dec 28 11:39:04 (I) Making day1 and weekly snapshot GZIPs
Dec 28 11:39:04 (I) Reading dataset list from stdin ..
Dec 28 11:39:04 (I) Making GZIP for rhqbbar
Dec 28 11:39:04 (I) /nexsanpool2/rhqbbar-day-1-20151201.gz already exists
Dec 28 11:39:04 (I) /nexsanpool2/rhqbbar-week-1-20151207.gz already exists
Dec 28 11:39:04 (I) Creating /nexsanpool2/rhqbbar-week-2-20151214.gz with: zfs send -i nexsanpool2/rhqbbar@week-1-20151207 nexsanpool2/rhqbbar@week-2-20151214 | gzip > /nexsanpool2/rhqbbar-week-2-20151214.gz &
Dec 28 11:39:04 (I) Creating /nexsanpool2/rhqbbar-week-3-20151221.gz with: zfs send -i nexsanpool2/rhqbbar@week-2-20151214 nexsanpool2/rhqbbar@week-3-20151221 | gzip > /nexsanpool2/rhqbbar-week-3-20151221.gz &

[1]+  Done                    nohup echo rhqbbar | ./weekly-zfs-gzip.sh
[root@dslustre21 zfs.gba]# 

After a little while I see all the gzips that should are done, and the processes where running with nohup and in background (so it is safe if my ssh session dies) :)

[root@dslustre21 zfs.gba]# ls -lah /nexsanpool2/rhqbbar*.gz
-rw-r--r-- 1 root root 657G Dec 11 08:27 /nexsanpool2/rhqbbar-day-1-20151201.gz
-rw-r--r-- 1 root root  148 Dec 10 14:15 /nexsanpool2/rhqbbar-week-1-20151207.gz
-rw-r--r-- 1 root root  147 Dec 28 11:39 /nexsanpool2/rhqbbar-week-2-20151214.gz
-rw-r--r-- 1 root root 8.0G Dec 28 11:56 /nexsanpool2/rhqbbar-week-3-20151221.gz
[root@dslustre21 zfs.gba]# 

#16 Updated by Gerard Bernabeu Altayo almost 5 years ago

A way to run this manually is:


[root@dslustre21 zfs.gba]# nohup echo rhqbbar | ./weekly-zfs-gzip.sh >> /tmp/gba.log & 2>&1
[1] 18127
[root@dslustre21 zfs.gba]# nohup: ignoring input and redirecting stderr to stdout

[1]+  Done                    nohup echo rhqbbar | ./weekly-zfs-gzip.sh >> /tmp/gba.log
[root@dslustre21 zfs.gba]# 

Then the log shows:

[root@dslustre21 zfs.gba]# cat  /tmp/gba.log
Dec 28 12:15:24 (I) Making day1 and weekly snapshot GZIPs
Dec 28 12:15:24 (I) Reading dataset list from stdin ..
Dec 28 12:15:24 (I) Making GZIP for rhqbbar
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-day-1-20151201.gz already exists
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-week-1-20151207.gz already exists
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-week-2-20151214.gz already exists
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-week-3-20151221.gz already exists

This is the last step that can be ran from dslustre21, I'm actually thinking on running it from the same script... Even if it puts some load on nexsanpool2, it may be worth it as it will only succeed the day after the new weeklys are done and it will never trigger an error...

#17 Updated by Gerard Bernabeu Altayo almost 5 years ago

The script is done, I ended up mixing it all in one, running now for all datasets. This will generate all the daily replicas, the weekly copies and the needed gzips. It will do each stage at once, giving enough time for one 'wave' to kick in before the next one does.

I am not replacing the original backup script yet as I want to test it thoroughly first. The only thing that makes me nervous about adding this with the daily snapshot script is that this script could take a long time to finish and we don't want the daily snapshot to be delayed because of this.

Also this script is still missing the locking mechanism (that I should copy from other scripts I have).

[root@dslustre21 zfs.gba]# nohup cat /root/zfs-scripts/dataset-list-reduced | /root/zfs.gba/replica-zfs-snapshot.sh >> /tmp/gba.log 2>&1 &
[1] 27272
[root@dslustre21 zfs.gba]# nohup: ignoring input and redirecting stderr to stdout

And here is the output:

[root@dslustre21 zfs.gba]# nohup: ignoring input and redirecting stderr to stdout

[1]+  Done                    nohup cat /root/zfs-scripts/dataset-list-reduced | /root/zfs.gba/replica-zfs-snapshot.sh >> /tmp/gba.log 2>&1
[root@dslustre21 zfs.gba]# cat /tmp/gba.log
Dec 28 12:15:24 (I) Making day1 and weekly snapshot GZIPs
Dec 28 12:15:24 (I) Reading dataset list from stdin ..
Dec 28 12:15:24 (I) Making GZIP for rhqbbar
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-day-1-20151201.gz already exists
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-week-1-20151207.gz already exists
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-week-2-20151214.gz already exists
Dec 28 12:15:24 (I) /nexsanpool2/rhqbbar-week-3-20151221.gz already exists
Dec 28 13:37:21 (I) Reading dataset list from stdin ..
Dec 28 13:37:21 (I) Making daily snapshots - Snapshot date tag is 28

Dec 28 13:37:21 (I) Replicating snapshots for axial
Dec 28 13:37:21 (I) Replicating snapshots for charmonium
Dec 28 13:37:22 (I) Replicating snapshots for c51
Dec 28 13:37:22 (I) Replicating snapshots for disco
Dec 28 13:37:22 (I) Replicating snapshots for fourpluseight
Dec 28 13:37:22 (I) Replicating snapshots for heavylight
Dec 28 13:37:22 (I) Replicating snapshots for hpqcd
Dec 28 13:37:22 (I) Replicating snapshots for mixbk
Dec 28 13:37:22 (I) Replicating snapshots for nonperturbativestaggered
Dec 28 13:37:22 (I) Replicating snapshots for ok-action
Dec 28 13:37:22 (I) Replicating snapshots for rbcdwf
Dec 28 13:37:22 (I) Replicating snapshots for rhqbbar
Dec 28 13:37:22 (I) Making weekly snapshots
Dec 28 13:37:22 (I) Checking weekly snapshots for axial
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:22 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/axial@day-21-20151221 nexsanpool1/axial@day-28-20151228 | zfs recv nexsanpool2/axial@week-4-20151228
WARNING: could not send nexsanpool1/axial@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:22 (E) Failed (256) to execute: 

Dec 28 13:37:22 (I) Checking weekly snapshots for charmonium
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:22 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/charmonium@day-21-20151221 nexsanpool1/charmonium@day-28-20151228 | zfs recv nexsanpool2/charmonium@week-4-20151228
WARNING: could not send nexsanpool1/charmonium@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:22 (E) Failed (256) to execute: 

Dec 28 13:37:22 (I) Checking weekly snapshots for c51
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:22 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/c51@day-21-20151221 nexsanpool1/c51@day-28-20151228 | zfs recv nexsanpool2/c51@week-4-20151228
WARNING: could not send nexsanpool1/c51@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for disco
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/disco@day-21-20151221 nexsanpool1/disco@day-28-20151228 | zfs recv nexsanpool2/disco@week-4-20151228
WARNING: could not send nexsanpool1/disco@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for fourpluseight
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/fourpluseight@day-21-20151221 nexsanpool1/fourpluseight@day-28-20151228 | zfs recv nexsanpool2/fourpluseight@week-4-20151228
WARNING: could not send nexsanpool1/fourpluseight@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for heavylight
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/heavylight@day-21-20151221 nexsanpool1/heavylight@day-28-20151228 | zfs recv nexsanpool2/heavylight@week-4-20151228
WARNING: could not send nexsanpool1/heavylight@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for hpqcd
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/hpqcd@day-21-20151221 nexsanpool1/hpqcd@day-28-20151228 | zfs recv nexsanpool2/hpqcd@week-4-20151228
WARNING: could not send nexsanpool1/hpqcd@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for mixbk
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/mixbk@day-21-20151221 nexsanpool1/mixbk@day-28-20151228 | zfs recv nexsanpool2/mixbk@week-4-20151228
WARNING: could not send nexsanpool1/mixbk@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for nonperturbativestaggered
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/nonperturbativestaggered@day-21-20151221 nexsanpool1/nonperturbativestaggered@day-28-20151228 | zfs recv nexsanpool2/nonperturbativestaggered@week-4-20151228
WARNING: could not send nexsanpool1/nonperturbativestaggered@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for ok-action
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/ok-action@day-21-20151221 nexsanpool1/ok-action@day-28-20151228 | zfs recv nexsanpool2/ok-action@week-4-20151228
WARNING: could not send nexsanpool1/ok-action@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for rbcdwf
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/rbcdwf@day-21-20151221 nexsanpool1/rbcdwf@day-28-20151228 | zfs recv nexsanpool2/rbcdwf@week-4-20151228
WARNING: could not send nexsanpool1/rbcdwf@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Checking weekly snapshots for rhqbbar
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-4-20151228
Dec 28 13:37:23 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/rhqbbar@day-21-20151221 nexsanpool1/rhqbbar@day-28-20151228 | zfs recv nexsanpool2/rhqbbar@week-4-20151228
WARNING: could not send nexsanpool1/rhqbbar@day-28-20151228: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Dec 28 13:37:23 (E) Failed (256) to execute: 

Dec 28 13:37:23 (I) Making GZIPs
Dec 28 13:37:23 (I) Making GZIP for axial
Dec 28 13:37:23 (I) /nexsanpool2/axial-day-1-20151201.gz already exists
Dec 28 13:37:23 (I) /nexsanpool2/axial-week-1-20151207.gz already exists
Dec 28 13:37:23 (I) Creating /nexsanpool2/axial-week-2-20151214.gz with: zfs send -i nexsanpool2/axial@week-1-20151207 nexsanpool2/axial@week-2-20151214 | gzip > /nexsanpool2/axial-week-2-20151214.gz &
Dec 28 13:37:23 (I) Creating /nexsanpool2/axial-week-3-20151221.gz with: zfs send -i nexsanpool2/axial@week-2-20151214 nexsanpool2/axial@week-3-20151221 | gzip > /nexsanpool2/axial-week-3-20151221.gz &
Dec 28 13:37:23 (I) Making GZIP for charmonium
Dec 28 13:37:23 (I) /nexsanpool2/charmonium-day-1-20151201.gz already exists
Dec 28 13:37:23 (I) /nexsanpool2/charmonium-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/charmonium-week-2-20151214.gz with: zfs send -i nexsanpool2/charmonium@week-1-20151207 nexsanpool2/charmonium@week-2-20151214 | gzip > /nexsanpool2/charmonium-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/charmonium-week-3-20151221.gz with: zfs send -i nexsanpool2/charmonium@week-2-20151214 nexsanpool2/charmonium@week-3-20151221 | gzip > /nexsanpool2/charmonium-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for c51
Dec 28 13:37:24 (I) /nexsanpool2/c51-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/c51-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/c51-week-2-20151214.gz with: zfs send -i nexsanpool2/c51@week-1-20151207 nexsanpool2/c51@week-2-20151214 | gzip > /nexsanpool2/c51-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/c51-week-3-20151221.gz with: zfs send -i nexsanpool2/c51@week-2-20151214 nexsanpool2/c51@week-3-20151221 | gzip > /nexsanpool2/c51-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for disco
Dec 28 13:37:24 (I) /nexsanpool2/disco-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/disco-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/disco-week-2-20151214.gz with: zfs send -i nexsanpool2/disco@week-1-20151207 nexsanpool2/disco@week-2-20151214 | gzip > /nexsanpool2/disco-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/disco-week-3-20151221.gz with: zfs send -i nexsanpool2/disco@week-2-20151214 nexsanpool2/disco@week-3-20151221 | gzip > /nexsanpool2/disco-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for fourpluseight
Dec 28 13:37:24 (I) /nexsanpool2/fourpluseight-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/fourpluseight-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/fourpluseight-week-2-20151214.gz with: zfs send -i nexsanpool2/fourpluseight@week-1-20151207 nexsanpool2/fourpluseight@week-2-20151214 | gzip > /nexsanpool2/fourpluseight-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/fourpluseight-week-3-20151221.gz with: zfs send -i nexsanpool2/fourpluseight@week-2-20151214 nexsanpool2/fourpluseight@week-3-20151221 | gzip > /nexsanpool2/fourpluseight-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for heavylight
Dec 28 13:37:24 (I) /nexsanpool2/heavylight-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/heavylight-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/heavylight-week-2-20151214.gz with: zfs send -i nexsanpool2/heavylight@week-1-20151207 nexsanpool2/heavylight@week-2-20151214 | gzip > /nexsanpool2/heavylight-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/heavylight-week-3-20151221.gz with: zfs send -i nexsanpool2/heavylight@week-2-20151214 nexsanpool2/heavylight@week-3-20151221 | gzip > /nexsanpool2/heavylight-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for hpqcd
Dec 28 13:37:24 (I) /nexsanpool2/hpqcd-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/hpqcd-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/hpqcd-week-2-20151214.gz with: zfs send -i nexsanpool2/hpqcd@week-1-20151207 nexsanpool2/hpqcd@week-2-20151214 | gzip > /nexsanpool2/hpqcd-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/hpqcd-week-3-20151221.gz with: zfs send -i nexsanpool2/hpqcd@week-2-20151214 nexsanpool2/hpqcd@week-3-20151221 | gzip > /nexsanpool2/hpqcd-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for mixbk
Dec 28 13:37:24 (I) /nexsanpool2/mixbk-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/mixbk-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/mixbk-week-2-20151214.gz with: zfs send -i nexsanpool2/mixbk@week-1-20151207 nexsanpool2/mixbk@week-2-20151214 | gzip > /nexsanpool2/mixbk-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/mixbk-week-3-20151221.gz with: zfs send -i nexsanpool2/mixbk@week-2-20151214 nexsanpool2/mixbk@week-3-20151221 | gzip > /nexsanpool2/mixbk-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for nonperturbativestaggered
Dec 28 13:37:24 (I) /nexsanpool2/nonperturbativestaggered-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/nonperturbativestaggered-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz with: zfs send -i nexsanpool2/nonperturbativestaggered@week-1-20151207 nexsanpool2/nonperturbativestaggered@week-2-20151214 | gzip > /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz with: zfs send -i nexsanpool2/nonperturbativestaggered@week-2-20151214 nexsanpool2/nonperturbativestaggered@week-3-20151221 | gzip > /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for ok-action
Dec 28 13:37:24 (I) /nexsanpool2/ok-action-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/ok-action-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/ok-action-week-2-20151214.gz with: zfs send -i nexsanpool2/ok-action@week-1-20151207 nexsanpool2/ok-action@week-2-20151214 | gzip > /nexsanpool2/ok-action-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/ok-action-week-3-20151221.gz with: zfs send -i nexsanpool2/ok-action@week-2-20151214 nexsanpool2/ok-action@week-3-20151221 | gzip > /nexsanpool2/ok-action-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for rbcdwf
Dec 28 13:37:24 (I) /nexsanpool2/rbcdwf-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/rbcdwf-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) Creating /nexsanpool2/rbcdwf-week-2-20151214.gz with: zfs send -i nexsanpool2/rbcdwf@week-1-20151207 nexsanpool2/rbcdwf@week-2-20151214 | gzip > /nexsanpool2/rbcdwf-week-2-20151214.gz &
Dec 28 13:37:24 (I) Creating /nexsanpool2/rbcdwf-week-3-20151221.gz with: zfs send -i nexsanpool2/rbcdwf@week-2-20151214 nexsanpool2/rbcdwf@week-3-20151221 | gzip > /nexsanpool2/rbcdwf-week-3-20151221.gz &
Dec 28 13:37:24 (I) Making GZIP for rhqbbar
Dec 28 13:37:24 (I) /nexsanpool2/rhqbbar-day-1-20151201.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/rhqbbar-week-1-20151207.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/rhqbbar-week-2-20151214.gz already exists
Dec 28 13:37:24 (I) /nexsanpool2/rhqbbar-week-3-20151221.gz already exists
[root@dslustre21 zfs.gba]# 

All the weeklies have failed because today is 28th but the 28th daily is not created yet, that's fine as it will succeed tomorrow or also it would today if this was executed as designed, after the daily snapshot replication!

This Error can actually be used to trigger a human to look into the script because it should never exist (and if other snapshots don't exist we will not be warned by design!).

Now if we look at what's going on at the system level, the gzips are kicking in:

[root@dslustre21 zfs.gba]# ps faux | grep -B1 gzip
root     28646  5.0  0.0 108940  1908 pts/2    R+   13:44   0:00  |       \_ ps faux
root     28647  0.0  0.0 103340  1000 pts/2    S+   13:44   0:00  |       \_ grep -B1 gzip
--
root     27592  0.9  0.0 126032  1372 pts/2    D    13:37   0:04 zfs send -i nexsanpool2/heavylight week-2-20151214 nexsanpool2/heavylight week-3-20151221
root     27593 99.2  0.0   4292   656 pts/2    R    13:37   6:41 gzip
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# ls -lah /nexsanpool2/*.gz
-rw-r--r-- 1 root root 8.1G Dec 10 14:30 /nexsanpool2/axial-day-1-20151201.gz
-rw-r--r-- 1 root root 1.8M Dec 10 14:15 /nexsanpool2/axial-week-1-20151207.gz
-rw-r--r-- 1 root root  146 Dec 28 13:37 /nexsanpool2/axial-week-2-20151214.gz
-rw-r--r-- 1 root root  145 Dec 28 13:37 /nexsanpool2/axial-week-3-20151221.gz
-rw-r--r-- 1 root root 563M Dec 10 14:07 /nexsanpool2/c51-day-1-20151201.gz
-rw-r--r-- 1 root root  142 Dec 10 14:15 /nexsanpool2/c51-week-1-20151207.gz
-rw-r--r-- 1 root root  143 Dec 28 13:37 /nexsanpool2/c51-week-2-20151214.gz
-rw-r--r-- 1 root root  144 Dec 28 13:37 /nexsanpool2/c51-week-3-20151221.gz
-rw-r--r-- 1 root root 2.7K Dec 10 14:06 /nexsanpool2/charmonium-day-1-20151201.gz
-rw-r--r-- 1 root root  151 Dec 10 14:15 /nexsanpool2/charmonium-week-1-20151207.gz
-rw-r--r-- 1 root root  151 Dec 28 13:37 /nexsanpool2/charmonium-week-2-20151214.gz
-rw-r--r-- 1 root root  151 Dec 28 13:37 /nexsanpool2/charmonium-week-3-20151221.gz
-rw-r--r-- 1 root root 1.9G Dec 10 14:22 /nexsanpool2/disco-day-1-20151201.gz
-rw-r--r-- 1 root root  146 Dec 10 14:15 /nexsanpool2/disco-week-1-20151207.gz
-rw-r--r-- 1 root root  146 Dec 28 13:37 /nexsanpool2/disco-week-2-20151214.gz
-rw-r--r-- 1 root root  147 Dec 28 13:37 /nexsanpool2/disco-week-3-20151221.gz
-rw-r--r-- 1 root root 7.0G Dec 10 14:46 /nexsanpool2/fourpluseight-day-1-20151201.gz
-rw-r--r-- 1 root root  154 Dec 10 14:15 /nexsanpool2/fourpluseight-week-1-20151207.gz
-rw-r--r-- 1 root root  154 Dec 28 13:37 /nexsanpool2/fourpluseight-week-2-20151214.gz
-rw-r--r-- 1 root root 275M Dec 28 13:39 /nexsanpool2/fourpluseight-week-3-20151221.gz
-rw-r--r-- 1 root root 2.1T Dec 14 14:35 /nexsanpool2/heavylight-day-1-20151201.gz
-rw-r--r-- 1 root root 7.6G Dec 10 14:29 /nexsanpool2/heavylight-week-1-20151207.gz
-rw-r--r-- 1 root root 4.2G Dec 28 13:43 /nexsanpool2/heavylight-week-2-20151214.gz
-rw-r--r-- 1 root root 6.2G Dec 28 13:45 /nexsanpool2/heavylight-week-3-20151221.gz
-rw-r--r-- 1 root root 133G Dec 10 22:10 /nexsanpool2/hpqcd-day-1-20151201.gz
-rw-r--r-- 1 root root  146 Dec 10 14:15 /nexsanpool2/hpqcd-week-1-20151207.gz
-rw-r--r-- 1 root root  146 Dec 28 13:37 /nexsanpool2/hpqcd-week-2-20151214.gz
-rw-r--r-- 1 root root 273M Dec 28 13:39 /nexsanpool2/hpqcd-week-3-20151221.gz
-rw-r--r-- 1 root root 135G Dec 11 04:10 /nexsanpool2/mixbk-day-1-20151201.gz
-rw-r--r-- 1 root root  146 Dec 10 14:15 /nexsanpool2/mixbk-week-1-20151207.gz
-rw-r--r-- 1 root root  146 Dec 28 13:37 /nexsanpool2/mixbk-week-2-20151214.gz
-rw-r--r-- 1 root root  146 Dec 28 13:37 /nexsanpool2/mixbk-week-3-20151221.gz
-rw-r--r-- 1 root root  19G Dec 10 15:35 /nexsanpool2/nonperturbativestaggered-day-1-20151201.gz
-rw-r--r-- 1 root root  165 Dec 10 14:15 /nexsanpool2/nonperturbativestaggered-week-1-20151207.gz
-rw-r--r-- 1 root root  164 Dec 28 13:37 /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz
-rw-r--r-- 1 root root  165 Dec 28 13:37 /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz
-rw-r--r-- 1 root root 5.4G Dec 10 14:29 /nexsanpool2/ok-action-day-1-20151201.gz
-rw-r--r-- 1 root root  150 Dec 10 14:15 /nexsanpool2/ok-action-week-1-20151207.gz
-rw-r--r-- 1 root root  149 Dec 28 13:37 /nexsanpool2/ok-action-week-2-20151214.gz
-rw-r--r-- 1 root root  150 Dec 28 13:37 /nexsanpool2/ok-action-week-3-20151221.gz
-rw-r--r-- 1 root root 474G Dec 11 03:10 /nexsanpool2/rbcdwf-day-1-20151201.gz
-rw-r--r-- 1 root root  148 Dec 10 14:15 /nexsanpool2/rbcdwf-week-1-20151207.gz
-rw-r--r-- 1 root root  147 Dec 28 13:37 /nexsanpool2/rbcdwf-week-2-20151214.gz
-rw-r--r-- 1 root root  147 Dec 28 13:37 /nexsanpool2/rbcdwf-week-3-20151221.gz
-rw-r--r-- 1 root root 657G Dec 11 08:27 /nexsanpool2/rhqbbar-day-1-20151201.gz
-rw-r--r-- 1 root root  148 Dec 10 14:15 /nexsanpool2/rhqbbar-week-1-20151207.gz
-rw-r--r-- 1 root root  147 Dec 28 11:39 /nexsanpool2/rhqbbar-week-2-20151214.gz
-rw-r--r-- 1 root root 8.0G Dec 28 11:56 /nexsanpool2/rhqbbar-week-3-20151221.gz
[root@dslustre21 zfs.gba]# 

And even while running a 2nd run of the script is harmless because the gzips are have been created already. This is telling me it is actually safe to run this script on a daily basis attached to the daily cron...

#18 Updated by Gerard Bernabeu Altayo over 4 years ago

montly backups worked fine with a manual run of:

nohup cat /root/zfs-scripts/dataset-list-reduced | /root/zfs.gba/replica-zfs-snapshot.sh >> /tmp/gba2.log 2>&1 &

Looking at automating the backups to tape.

#19 Updated by Gerard Bernabeu Altayo over 4 years ago

I've tested the script and it works as designed:

1. It complains if there are less than 60 files to copy
2. It complains if there are 60 files but some of them have been modified over the last hour

[root@lqcdsrm ~]# /home/gerard1/backups_to_tape.sh
ERROR: gzips may still be in creation. Check because [ 59 -ne 59 ] || [ 59 -ne 60 ]
[root@lqcdsrm ~]# echo $?
2
[root@lqcdsrm ~]# /home/gerard1/backups_to_tape.sh
ERROR: gzips may still be in creation. Check because [ 60 -ne 59 ] || [ 59 -ne 60 ]
[root@lqcdsrm ~]#

Now I'll wait a little bit and trigger it again with:

[root@lqcdsrm ~]# nohup /home/gerard1/backups_to_tape.sh &
[1] 31082
[root@lqcdsrm ~]# nohup: appending output to `nohup.out'

[1]+  Exit 2                  nohup /home/gerard1/backups_to_tape.sh
[root@lqcdsrm ~]# cat nohup.out 
Copying 59 /tmp/tapebackuplist.591 files
echo log=/var/log/lqcd-backup/encp_bkp...2015-12-30--101338.log
ENCP=encp --verbose 2 --delayed-dismount 2 --bypass-filesystem-max-filesize-check --buffer-size 1048576 --threaded
ERROR: gzips may still be in creation. Check because [ 60 -ne 59 ] || [ 59 -ne 60 ]
[root@lqcdsrm ~]# 

#20 Updated by Gerard Bernabeu Altayo over 4 years ago

Now making the copies for real:

[root@lqcdsrm ~]# rm nohup.out 
rm: remove regular file `nohup.out'? yes
[root@lqcdsrm ~]# nohup /home/gerard1/backups_to_tape.sh &
[1] 7620
[root@lqcdsrm ~]# nohup: appending output to `nohup.out'

[root@lqcdsrm ~]# cat nohup.out 
Copying 60 /tmp/tapebackuplist.621 files
echo log=/var/log/lqcd-backup/encp_bkp.2015-12-30--113245.log
ENCP=encp --verbose 2 --delayed-dismount 2 --bypass-filesystem-max-filesize-check --buffer-size 1048576 --threaded
[root@lqcdsrm ~]# cat /var/log/lqcd-backup/encp_bkp.2015-12-30--113245.log
dst=/pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30
filelist=/tmp/tapebackuplist.621
mkdir: cannot create directory `/pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30': File exists
encp:   started:   Wed Dec 30 11:32:45 CST 2015
file: /nexsanpool2/axial-day-1-20151201.gz
Start time: Wed Dec 30 11:32:45 2015
User: root(0)  Group: root(0)  Euser: root(0)  Egroup: root(0)
Command line: encp --verbose 2 --delayed-dismount 2 --bypass-filesystem-max-filesize-check --buffer-size 1048576 --threaded /nexsanpool2/axial-day-1-20151201.gz /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/
Version: v3_11 CVS $Revision: 1.990.2.23 $ <frozen>
OS: Linux 2.6.18-348.6.1.el5 x86_64 Release:  Scientific Linux SLF release 5.2 (Lederman)
Library: CD-10KDF1  Storage Group: lqcd  File Family: project_backup_B  FF Wrapper: cern  FF Width: 1
Current working directory: lqcdsrm.fnal.gov:/root
[root@lqcdsrm ~]# 

And I'll leave the processes running:

[root@lqcdsrm ~]# ps faux | tail -6
root     10887  0.0  0.0  72316  3344 ?        Ss   09:45   0:00  \_ sshd: root@pts/2 
root     10888  0.0  0.0  66208  1700 pts/2    Ss   09:45   0:00      \_ -bash
root      7620  0.0  0.0  65948  1324 pts/2    S    11:32   0:00          \_ /bin/bash /home/gerard1/backups_to_tape.sh
root      7740  0.1  0.0  15316  9492 pts/2    S    11:32   0:00          |   \_ encp --verbose 2 --delayed-dismount 2 --bypass-filesystem-max-filesize-check --buffer-size 1048576 --threaded /nexsanpool2/axial-day-1-20151201.gz /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/
root      7785  0.0  0.0  65860  1240 pts/2    R+   11:34   0:00          \_ ps faux
root      7786  0.0  0.0  58928   508 pts/2    S+   11:34   0:00          \_ tail -6
[root@lqcdsrm ~]# logout

[root@lqcdsrm ~]# ps faux | tail -6
root      7788  0.0  0.0  72316  3328 ?        Ss   11:34   0:00  \_ sshd: root@pts/0 
root      7789  0.6  0.0  66084  1624 pts/0    Ss   11:34   0:00      \_ -bash
root      7835  0.0  0.0  65860  1236 pts/0    R+   11:34   0:00          \_ ps faux
root      7836  0.0  0.0  58928   512 pts/0    S+   11:34   0:00          \_ tail -6
root      7620  0.0  0.0  65948  1324 ?        S    11:32   0:00 /bin/bash /home/gerard1/backups_to_tape.sh
root      7740  0.1  0.0  15316  9492 ?        S    11:32   0:00  \_ encp --verbose 2 --delayed-dismount 2 --bypass-filesystem-max-filesize-check --buffer-size 1048576 --threaded /nexsanpool2/axial-day-1-20151201.gz /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/
[root@lqcdsrm ~]# 

#21 Updated by Gerard Bernabeu Altayo over 4 years ago

After a while I can see how it's working fine:

[root@lqcdsrm ~]# ls -lah /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30
total 18G
drwxr-xr-x 28 root root  512 Dec 30 11:40 .
drwxr-xr-x  8 root root  512 Dec 30 10:08 ..
-rw-r--r--  1 root root 8.1G Dec 30 11:36 axial-day-1-20151201.gz
-rw-r--r--  1 root root 1.8M Dec 30 11:36 axial-week-1-20151207.gz
-rw-r--r--  1 root root  146 Dec 30 11:36 axial-week-2-20151214.gz
-rw-r--r--  1 root root  145 Dec 30 11:36 axial-week-3-20151221.gz
-rw-r--r--  1 root root 3.2M Dec 30 11:36 axial-week-4-20151228.gz
-rw-r--r--  1 root root 563M Dec 30 11:36 c51-day-1-20151201.gz
-rw-r--r--  1 root root  142 Dec 30 11:37 c51-week-1-20151207.gz
-rw-r--r--  1 root root  143 Dec 30 11:37 c51-week-2-20151214.gz
-rw-r--r--  1 root root  144 Dec 30 11:37 c51-week-3-20151221.gz
-rw-r--r--  1 root root  145 Dec 30 11:37 c51-week-4-20151228.gz
-rw-r--r--  1 root root 2.7K Dec 30 11:37 charmonium-day-1-20151201.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-1-20151207.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-2-20151214.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-3-20151221.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-4-20151228.gz
-rw-r--r--  1 root root 1.9G Dec 30 11:38 disco-day-1-20151201.gz
-rw-r--r--  1 root root  146 Dec 30 11:38 disco-week-1-20151207.gz
-rw-r--r--  1 root root  146 Dec 30 11:38 disco-week-2-20151214.gz
-rw-r--r--  1 root root  147 Dec 30 11:38 disco-week-3-20151221.gz
-rw-r--r--  1 root root  146 Dec 30 11:38 disco-week-4-20151228.gz
-rw-r--r--  1 root root 7.0G Dec 30 11:40 fourpluseight-day-1-20151201.gz
-rw-r--r--  1 root root  154 Dec 30 11:40 fourpluseight-week-1-20151207.gz
-rw-r--r--  1 root root  154 Dec 30 11:40 fourpluseight-week-2-20151214.gz
-rw-r--r--  1 root root 275M Dec 30 11:40 fourpluseight-week-3-20151221.gz
-rw-r--r--  1 root root  154 Dec 30 11:40 fourpluseight-week-4-20151228.gz
-rw-r--r--  1 root root    0 Dec 30 11:40 heavylight-day-1-20151201.gz
[root@lqcdsrm ~]# 

Once this is done I need to figure out how the cleanup process works. I will make a proposal to Amitoj before executing it because we did not go through it together.

For now the new procedure is:

1. Daily: [root@dslustre21 ]# cat /root/zfs-scripts/dataset-list-reduced | /root/zfs.gba/replica-zfs-snapshot.sh
2. Every first day of the month: [root@lqcdsrm ~]# nohup /home/gerard1/backups_to_tape.sh &

TODO:

- Improve /home/gerard1/backups_to_tape.sh so that it can be executed N times without causing trouble
- Make a cleanup script that:
1. Removes gzips (after checking they're all good)
2. Removes nexsanpool1/2 mirrored images
3. Creates day1, should try to do it in an efficient way by using http://www.freebsddiary.org/zfs-promote.php and later on trying to apply a diff to it.

#22 Updated by Gerard Bernabeu Altayo over 4 years ago

All backed up files have been successfully copied to enstore:

[root@lqcdsrm ~]# cat /var/log/lqcd-backup/encp_bkp.2015-12-30--113245.log | grep -c 'Return code 0'
60
[root@lqcdsrm ~]# cat /var/log/lqcd-backup/encp_bkp.2015-12-30--113245.log | grep  'Return code 0' | head -1
Return code 0 /nexsanpool2/axial-day-1-20151201.gz
[root@lqcdsrm ~]# cat /var/log/lqcd-backup/encp_bkp.2015-12-30--113245.log | grep  'Return code 0' | tail -1
Return code 0 /nexsanpool2/rhqbbar-week-4-20151228.gz
[root@lqcdsrm ~]# 

Another check:

[root@lqcdsrm ~]# ls /pnfs/lqcd/backups/lqcdproj_backups/project_backup_*/2015-12-30 | wc -l
60
[root@lqcdsrm ~]# 

#23 Updated by Gerard Bernabeu Altayo over 4 years ago

Now I have a script that will initialize every month once it's sure that all the gzips are done, I've hacked it a bit for it to give me the right commands for the axial dataset (I messed up with it and need to rebuild week-5 and send it to tape!).

[root@dslustre21 zfs.gba]# echo axial | ./first-of-month.pl  
Jan 05 08:38:47 (I) Reading dataset list from stdin ..
Jan 05 08:38:47 (I) Checking snapshots for axial
Jan 05 08:38:47 (I) /nexsanpool2/axial-day-1-20151201.gz already exists and was last modified 2225290 seconds ago
Jan 05 08:38:47 (I) /nexsanpool2/axial-week-1-20151207.gz already exists and was last modified 2226209 seconds ago
Jan 05 08:38:47 (I) /nexsanpool2/axial-week-2-20151214.gz already exists and was last modified 673284 seconds ago
Jan 05 08:38:47 (I) /nexsanpool2/axial-week-3-20151221.gz already exists and was last modified 673284 seconds ago
Jan 05 08:38:47 (I) /nexsanpool2/axial-week-4-20151228.gz already exists and was last modified 604523 seconds ago
Jan 05 08:38:47 (E) /nexsanpool2/axial-week-5-20151231.gz DOES NOT exist or was modified during the last 5 minutes
Jan 05 08:38:47 (I) zfs destroy -vr nexsanpool2/axial
Jan 05 08:38:47 (I) zfs destroy -vr nexsanpool1/axial
Jan 05 08:38:47 (I) zfs send -v projectzfspool/projectzfs/axial@day-1-20160101 | zfs recv nexsanpool1/axial
Jan 05 08:38:47 (I) zfs send -v nexsanpool1/axial@day-1-20160101 | zfs recv nexsanpool2/axial
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-1-20151201
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-2-20151202
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-3-20151203
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-4-20151204
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-5-20151205
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-6-20151206
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-7-20151207
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-8-20151208
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-9-20151209
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-10-20151210
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-11-20151211
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-12-20151212
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-13-20151213
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-14-20151214
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-15-20151215
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-16-20151216
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-17-20151217
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-18-20151218
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-19-20151219
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-20-20151220
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-21-20151221
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-22-20151222
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-23-20151223
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-24-20151224
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-25-20151225
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-26-20151226
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-27-20151227
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-28-20151228
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-29-20151229
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-30-20151230
Jan 05 08:38:47 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-31-20151231
Jan 05 08:38:47 (E) NOT initializing for axial because missingsnaps=6 && day-1-20160101=1 && day-1-20151201=1
[root@dslustre21 zfs.gba]# 

Doing it (a bit changed to do December'15):

[root@dslustre21 zfs.gba]# zfs destroy -vr nexsanpool2/axial
will destroy nexsanpool2/axial@day-2-20160102
will destroy nexsanpool2/axial@day-3-20160103
will destroy nexsanpool2/axial@day-1-20160101
will destroy nexsanpool2/axial
[root@dslustre21 zfs.gba]# zfs send -v nexsanpool1/axial@day-1-20151201 | zfs recv nexsanpool2/axial
send from @ to nexsanpool1/axial@day-1-20151201 estimated size is 14.4G
total estimated size is 14.4G
TIME        SENT   SNAPSHOT
08:40:26   57.3M   nexsanpool1/axial@day-1-20151201
08:40:27    196M   nexsanpool1/axial@day-1-20151201
...
08:42:28   14.6G   nexsanpool1/axial@day-1-20151201
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep axial | grep nexsanpool2
nexsanpool2/axial@day-1-20151201                                        0      -  12.0G  -
[root@dslustre21 zfs.gba]# 

Now I'm regenerating all the weeklies, will have to hack the script first!

[root@dslustre21 zfs.gba]# vim replica-zfs-snapshot.pl 
diff --git a/replica-zfs-snapshot.pl b/replica-zfs-snapshot.pl
index 44dfb48..efe6e65 100755
--- a/replica-zfs-snapshot.pl
+++ b/replica-zfs-snapshot.pl
@@ -5,16 +5,17 @@
 # within /root/zfs-scripts/daily-zfs-snapshot

 ##simulating 2015-12-31:
-#$timedate = '20151231';
-#$day = '31';
-#$timemonth = '201512';
+$timedate = '20151231';
+$day = '31';
+$timemonth = '201512';

-$timedate = `/bin/date +"%Y%m%d"`;
-$day = `/bin/date +"%-d"`;
-$timemonth = `/bin/date +"%Y%m"`;
+#$timedate = `/bin/date +"%Y%m%d"`;
+#$day = `/bin/date +"%-d"`;
+#$timemonth = `/bin/date +"%Y%m"`;

[root@dslustre21 zfs.gba]# echo axial | ./replica-zfs-snapshot.pl
Jan 05 08:48:02 (I) Reading dataset list from stdin ..
Jan 05 08:48:02 (I) Making daily snapshots - Snapshot date tag is 31
Jan 05 08:48:02 (I) Replicating snapshots for axial
Jan 05 08:48:02 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/axial@day-3-20160103 projectzfspool/projectzfs/axial@day-4-20160104 | zfs recv nexsanpool1/axial@day-4-20160104
send from @day-3-20160103 to projectzfspool/projectzfs/axial@day-4-20160104 estimated size is 711M
total estimated size is 711M
TIME        SENT   SNAPSHOT
08:48:03   43.0M   projectzfspool/projectzfs/axial@day-4-20160104
08:48:04    128M   projectzfspool/projectzfs/axial@day-4-20160104
08:48:05    209M   projectzfspool/projectzfs/axial@day-4-20160104
08:48:06    292M   projectzfspool/projectzfs/axial@day-4-20160104
08:48:07    383M   projectzfspool/projectzfs/axial@day-4-20160104
08:48:08    490M   projectzfspool/projectzfs/axial@day-4-20160104
08:48:09    576M   projectzfspool/projectzfs/axial@day-4-20160104
08:48:10    666M   projectzfspool/projectzfs/axial@day-4-20160104
Jan 05 08:48:12 (I) Making weekly snapshots
Jan 05 08:48:12 (I) Checking weekly snapshots for axial
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create
 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Jan 05 08:48:12 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/axial@day-1-20151201 nexsanpool1/axial@day-7-20151207 | zfs recv nexsanpool2/axial@week-1-20151207
send from @day-1-20151201 to nexsanpool1/axial@day-7-20151207 estimated size is 5.39M
total estimated size is 5.39M
TIME        SENT   SNAPSHOT
Jan 05 08:48:13 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/axial@day-7-20151207 nexsanpool1/axial@day-14-20151214 | zfs recv nexsanpool2/axial@week-2-20151214
send from @day-7-20151207 to nexsanpool1/axial@day-14-20151214 estimated size is 512
total estimated size is 512
TIME        SENT   SNAPSHOT
Jan 05 08:48:14 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/axial@day-14-20151214 nexsanpool1/axial@day-21-20151221 | zfs recv nexsanpool2/axial@week-3-20151221
send from @day-14-20151214 to nexsanpool1/axial@day-21-20151221 estimated size is 512
total estimated size is 512
TIME        SENT   SNAPSHOT
Jan 05 08:48:14 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/axial@day-21-20151221 nexsanpool1/axial@day-28-20151228 | zfs recv nexsanpool2/axial@week-4-20151228
send from @day-21-20151221 to nexsanpool1/axial@day-28-20151228 estimated size is 10.1M
total estimated size is 10.1M
TIME        SENT   SNAPSHOT
08:48:15   8.00M   nexsanpool1/axial@day-28-20151228
Jan 05 08:48:16 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/axial@day-28-20151228 nexsanpool1/axial@day-31-20151231 | zfs recv nexsanpool2/axial@week-5-20151231
send from @day-28-20151228 to nexsanpool1/axial@day-31-20151231 estimated size is 10.3M
total estimated size is 10.3M
TIME        SENT   SNAPSHOT
08:48:17   12.0M   nexsanpool1/axial@day-31-20151231
Jan 05 08:48:17 (I) Making GZIPs
Jan 05 08:48:17 (I) Making GZIP for axial
Jan 05 08:48:18 (I) /nexsanpool2/axial-day-1-20151201.gz already exists
Jan 05 08:48:18 (I) /nexsanpool2/axial-week-1-20151207.gz already exists
Jan 05 08:48:18 (I) /nexsanpool2/axial-week-2-20151214.gz already exists
Jan 05 08:48:18 (I) /nexsanpool2/axial-week-3-20151221.gz already exists
Jan 05 08:48:18 (I) /nexsanpool2/axial-week-4-20151228.gz already exists
Jan 05 08:48:18 (I) Creating /nexsanpool2/axial-week-5-20151231.gz with: zfs send -i nexsanpool2/axial@week-4-20151228 nexsanpool2/axial@week-5-20151231 | gzip > /nexsanpool2/axial-week-5-20151231.gz &
[root@dslustre21 zfs.gba]# 

#24 Updated by Gerard Bernabeu Altayo over 4 years ago

With the hacked script that acts as if it was Dec 31st everything is ok:

[root@dslustre21 zfs.gba]# cat /root/zfs-scripts/dataset-list-reduced | ./replica-zfs-snapshot.pl 
Jan 05 11:07:07 (I) Reading dataset list from stdin ..
Jan 05 11:07:07 (I) Making daily snapshots - Snapshot date tag is 31
Jan 05 11:07:07 (I) Replicating snapshots for axial
Jan 05 11:07:07 (I) Replicating snapshots for charmonium
Jan 05 11:07:07 (I) Replicating snapshots for c51
Jan 05 11:07:07 (I) Replicating snapshots for disco
Jan 05 11:07:07 (I) Replicating snapshots for fourpluseight
Jan 05 11:07:08 (I) Replicating snapshots for heavylight
Jan 05 11:07:08 (I) Replicating snapshots for hpqcd
Jan 05 11:07:08 (I) Replicating snapshots for mixbk
Jan 05 11:07:08 (I) Replicating snapshots for nonperturbativestaggered
Jan 05 11:07:08 (I) Replicating snapshots for ok-action
Jan 05 11:07:08 (I) Replicating snapshots for rbcdwf
Jan 05 11:07:08 (I) Replicating snapshots for rhqbbar
Jan 05 11:07:08 (I) Making weekly snapshots
Jan 05 11:07:08 (I) Checking weekly snapshots for axial
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:08 (I) Found nexsanpool2/axial@week-5-20151231
Jan 05 11:07:08 (I) Checking weekly snapshots for charmonium
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:08 (I) Found nexsanpool2/charmonium@week-5-20151231
Jan 05 11:07:08 (I) Checking weekly snapshots for c51
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:08 (I) Found nexsanpool2/c51@week-5-20151231
Jan 05 11:07:08 (I) Checking weekly snapshots for disco
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:08 (I) Found nexsanpool2/disco@week-5-20151231
Jan 05 11:07:08 (I) Checking weekly snapshots for fourpluseight
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/fourpluseight@week-5-20151231
Jan 05 11:07:09 (I) Checking weekly snapshots for heavylight
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/heavylight@week-5-20151231
Jan 05 11:07:09 (I) Checking weekly snapshots for hpqcd
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/hpqcd@week-5-20151231
Jan 05 11:07:09 (I) Checking weekly snapshots for mixbk
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/mixbk@week-5-20151231
Jan 05 11:07:09 (I) Checking weekly snapshots for nonperturbativestaggered
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/nonperturbativestaggered@week-5-20151231
Jan 05 11:07:09 (I) Checking weekly snapshots for ok-action
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/ok-action@week-5-20151231
Jan 05 11:07:09 (I) Checking weekly snapshots for rbcdwf
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/rbcdwf@week-5-20151231
Jan 05 11:07:09 (I) Checking weekly snapshots for rhqbbar
There should exist
 day-1-20151201 week-1-20151207 week-2-20151214 week-3-20151221 week-4-20151228
Will create

Jan 05 11:07:09 (I) Found nexsanpool2/rhqbbar@week-5-20151231
Jan 05 11:07:09 (I) Making GZIPs
Jan 05 11:07:09 (I) Making GZIP for axial
Jan 05 11:07:09 (I) /nexsanpool2/axial-day-1-20151201.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/axial-week-1-20151207.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/axial-week-2-20151214.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/axial-week-3-20151221.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/axial-week-4-20151228.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/axial-week-5-20151231.gz already exists
Jan 05 11:07:09 (I) Making GZIP for charmonium
Jan 05 11:07:09 (I) /nexsanpool2/charmonium-day-1-20151201.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/charmonium-week-1-20151207.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/charmonium-week-2-20151214.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/charmonium-week-3-20151221.gz already exists
Jan 05 11:07:09 (I) /nexsanpool2/charmonium-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/charmonium-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for c51
Jan 05 11:07:10 (I) /nexsanpool2/c51-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/c51-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/c51-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/c51-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/c51-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/c51-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for disco
Jan 05 11:07:10 (I) /nexsanpool2/disco-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/disco-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/disco-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/disco-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/disco-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/disco-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for fourpluseight
Jan 05 11:07:10 (I) /nexsanpool2/fourpluseight-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/fourpluseight-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/fourpluseight-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/fourpluseight-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/fourpluseight-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/fourpluseight-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for heavylight
Jan 05 11:07:10 (I) /nexsanpool2/heavylight-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/heavylight-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/heavylight-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/heavylight-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/heavylight-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/heavylight-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for hpqcd
Jan 05 11:07:10 (I) /nexsanpool2/hpqcd-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/hpqcd-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/hpqcd-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/hpqcd-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/hpqcd-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/hpqcd-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for mixbk
Jan 05 11:07:10 (I) /nexsanpool2/mixbk-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/mixbk-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/mixbk-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/mixbk-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/mixbk-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/mixbk-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for nonperturbativestaggered
Jan 05 11:07:10 (I) /nexsanpool2/nonperturbativestaggered-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/nonperturbativestaggered-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/nonperturbativestaggered-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/nonperturbativestaggered-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/nonperturbativestaggered-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/nonperturbativestaggered-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for ok-action
Jan 05 11:07:10 (I) /nexsanpool2/ok-action-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/ok-action-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/ok-action-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/ok-action-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/ok-action-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/ok-action-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for rbcdwf
Jan 05 11:07:10 (I) /nexsanpool2/rbcdwf-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rbcdwf-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rbcdwf-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rbcdwf-week-3-20151221.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rbcdwf-week-4-20151228.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rbcdwf-week-5-20151231.gz already exists
Jan 05 11:07:10 (I) Making GZIP for rhqbbar
Jan 05 11:07:10 (I) /nexsanpool2/rhqbbar-day-1-20151201.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rhqbbar-week-1-20151207.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rhqbbar-week-2-20151214.gz already exists
Jan 05 11:07:10 (I) /nexsanpool2/rhqbbar-week-3-20151221.gz already exists
Jan 05 11:07:11 (I) /nexsanpool2/rhqbbar-week-4-20151228.gz already exists
Jan 05 11:07:11 (I) /nexsanpool2/rhqbbar-week-5-20151231.gz already exists
[root@dslustre21 zfs.gba]# echo $?
0
[root@dslustre21 zfs.gba]# 

Now copying to tape (with the hardcoded path so that copies go to the directory I did all the other backups):

[root@lqcdsrm ~]# /home/gerard1/backups_to_tape.sh
Copying 72 /tmp/tapebackuplist.866 files
echo log=/var/log/lqcd-backup/encp_bkp.2016-01-05--111011.log
ENCP=encp --verbose 2 --delayed-dismount 2 --bypass-filesystem-max-filesize-check --buffer-size 1048576 --threaded
[root@lqcdsrm ~]# ls -lah /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/
total 3.6T
drwxr-xr-x 75 root root  512 Jan  5 11:13 .
drwxr-xr-x  8 root root  512 Dec 30 10:08 ..
-rw-r--r--  1 root root 8.1G Dec 30 11:36 axial-day-1-20151201.gz
-rw-r--r--  1 root root 1.8M Dec 30 11:36 axial-week-1-20151207.gz
-rw-r--r--  1 root root  146 Dec 30 11:36 axial-week-2-20151214.gz
-rw-r--r--  1 root root  145 Dec 30 11:36 axial-week-3-20151221.gz
-rw-r--r--  1 root root 3.2M Dec 30 11:36 axial-week-4-20151228.gz
-rw-r--r--  1 root root 3.3M Jan  5 11:12 axial-week-5-20151231.gz
-rw-r--r--  1 root root 563M Dec 30 11:36 c51-day-1-20151201.gz
-rw-r--r--  1 root root  142 Dec 30 11:37 c51-week-1-20151207.gz
-rw-r--r--  1 root root  143 Dec 30 11:37 c51-week-2-20151214.gz
-rw-r--r--  1 root root  144 Dec 30 11:37 c51-week-3-20151221.gz
-rw-r--r--  1 root root  145 Dec 30 11:37 c51-week-4-20151228.gz
-rw-r--r--  1 root root  145 Jan  5 08:15 c51-week-5-20151231.gz
-rw-r--r--  1 root root 2.7K Dec 30 11:37 charmonium-day-1-20151201.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-1-20151207.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-2-20151214.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-3-20151221.gz
-rw-r--r--  1 root root  151 Dec 30 11:37 charmonium-week-4-20151228.gz
-rw-r--r--  1 root root  151 Jan  5 08:15 charmonium-week-5-20151231.gz
-rw-r--r--  1 root root 1.9G Dec 30 11:38 disco-day-1-20151201.gz
-rw-r--r--  1 root root  146 Dec 30 11:38 disco-week-1-20151207.gz
-rw-r--r--  1 root root  146 Dec 30 11:38 disco-week-2-20151214.gz
-rw-r--r--  1 root root  147 Dec 30 11:38 disco-week-3-20151221.gz
-rw-r--r--  1 root root  146 Dec 30 11:38 disco-week-4-20151228.gz
-rw-r--r--  1 root root  146 Jan  5 08:15 disco-week-5-20151231.gz
-rw-r--r--  1 root root 7.0G Dec 30 11:40 fourpluseight-day-1-20151201.gz
-rw-r--r--  1 root root  154 Dec 30 11:40 fourpluseight-week-1-20151207.gz
-rw-r--r--  1 root root  154 Dec 30 11:40 fourpluseight-week-2-20151214.gz
-rw-r--r--  1 root root 275M Dec 30 11:40 fourpluseight-week-3-20151221.gz
-rw-r--r--  1 root root  154 Dec 30 11:40 fourpluseight-week-4-20151228.gz
-rw-r--r--  1 root root  154 Jan  5 08:15 fourpluseight-week-5-20151231.gz
-rw-r--r--  1 root root 2.1T Dec 30 17:31 heavylight-day-1-20151201.gz
-rw-r--r--  1 root root 7.6G Dec 30 17:32 heavylight-week-1-20151207.gz
-rw-r--r--  1 root root 4.2G Dec 30 17:33 heavylight-week-2-20151214.gz
-rw-r--r--  1 root root  18G Dec 30 17:36 heavylight-week-3-20151221.gz
-rw-r--r--  1 root root  11G Dec 30 17:37 heavylight-week-4-20151228.gz
-rw-r--r--  1 root root  14G Jan  5 08:18 heavylight-week-5-20151231.gz
-rw-r--r--  1 root root 133G Dec 30 18:00 hpqcd-day-1-20151201.gz
-rw-r--r--  1 root root  146 Dec 30 18:01 hpqcd-week-1-20151207.gz
-rw-r--r--  1 root root  146 Dec 30 18:01 hpqcd-week-2-20151214.gz
-rw-r--r--  1 root root 273M Dec 30 18:01 hpqcd-week-3-20151221.gz
-rw-r--r--  1 root root  146 Dec 30 18:01 hpqcd-week-4-20151228.gz
-rw-r--r--  1 root root  146 Jan  5 08:18 hpqcd-week-5-20151231.gz
-rw-r--r--  1 root root 135G Dec 30 18:24 mixbk-day-1-20151201.gz
-rw-r--r--  1 root root  146 Dec 30 18:24 mixbk-week-1-20151207.gz
-rw-r--r--  1 root root  146 Dec 30 18:24 mixbk-week-2-20151214.gz
-rw-r--r--  1 root root  146 Dec 30 18:24 mixbk-week-3-20151221.gz
-rw-r--r--  1 root root  146 Dec 30 18:25 mixbk-week-4-20151228.gz
-rw-r--r--  1 root root  146 Jan  5 08:18 mixbk-week-5-20151231.gz
-rw-r--r--  1 root root  19G Dec 30 18:28 nonperturbativestaggered-day-1-20151201.gz
-rw-r--r--  1 root root  165 Dec 30 18:28 nonperturbativestaggered-week-1-20151207.gz
-rw-r--r--  1 root root  164 Dec 30 18:28 nonperturbativestaggered-week-2-20151214.gz
-rw-r--r--  1 root root  165 Dec 30 18:28 nonperturbativestaggered-week-3-20151221.gz
-rw-r--r--  1 root root  164 Dec 30 18:28 nonperturbativestaggered-week-4-20151228.gz
-rw-r--r--  1 root root  165 Jan  5 08:18 nonperturbativestaggered-week-5-20151231.gz
-rw-r--r--  1 root root 5.4G Dec 30 18:29 ok-action-day-1-20151201.gz
-rw-r--r--  1 root root  150 Dec 30 18:29 ok-action-week-1-20151207.gz
-rw-r--r--  1 root root  149 Dec 30 18:30 ok-action-week-2-20151214.gz
-rw-r--r--  1 root root  150 Dec 30 18:30 ok-action-week-3-20151221.gz
-rw-r--r--  1 root root  148 Dec 30 18:30 ok-action-week-4-20151228.gz
-rw-r--r--  1 root root  150 Jan  5 08:19 ok-action-week-5-20151231.gz
-rw-r--r--  1 root root 474G Dec 30 19:49 rbcdwf-day-1-20151201.gz
-rw-r--r--  1 root root  148 Dec 30 19:49 rbcdwf-week-1-20151207.gz
-rw-r--r--  1 root root  147 Dec 30 19:49 rbcdwf-week-2-20151214.gz
-rw-r--r--  1 root root  147 Dec 30 19:49 rbcdwf-week-3-20151221.gz
-rw-r--r--  1 root root  147 Dec 30 19:49 rbcdwf-week-4-20151228.gz
-rw-r--r--  1 root root  147 Jan  5 11:13 rbcdwf-week-5-20151231.gz
-rw-r--r--  1 root root 657G Dec 30 21:45 rhqbbar-day-1-20151201.gz
-rw-r--r--  1 root root  148 Dec 30 21:46 rhqbbar-week-1-20151207.gz
-rw-r--r--  1 root root  147 Dec 30 21:46 rhqbbar-week-2-20151214.gz
-rw-r--r--  1 root root 8.0G Dec 30 21:47 rhqbbar-week-3-20151221.gz
-rw-r--r--  1 root root  43M Dec 30 21:47 rhqbbar-week-4-20151228.gz
-rw-r--r--  1 root root  26M Jan  5 11:13 rhqbbar-week-5-20151231.gz
[root@lqcdsrm ~]# ls  /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/ | wc -l
72

It is all there :) I'll move on to do the turn to the new month (Jan'15).

#25 Updated by Gerard Bernabeu Altayo over 4 years ago

Trying to make it get to an error first:

[root@dslustre21 zfs.gba]# echo charmoniousdfa | ./first-of-month.pl  
Jan 05 11:21:18 (I) Reading dataset list from stdin ..
Jan 05 11:21:18 (I) Checking snapshots for charmoniousdfa
cannot open 'projectzfspool/projectzfs/charmoniousdfa': dataset does not exist
cannot open 'nexsanpool2/charmoniousdfa': dataset does not exist
Jan 05 11:21:18 (E) /nexsanpool2/charmoniousdfa-day-1-20151201.gz DOES NOT exist or was modified during the last 5 minutes
[root@dslustre21 zfs.gba]# echo $?
1

Now testing with an empty project:

[root@dslustre21 zfs.gba]# echo charmonium | ./first-of-month.pl  
Jan 05 11:22:16 (I) Reading dataset list from stdin ..
Jan 05 11:22:16 (I) Checking snapshots for charmonium
Jan 05 11:22:16 (I) /nexsanpool2/charmonium-day-1-20151201.gz already exists and was last modified 2236576 seconds ago
Jan 05 11:22:16 (I) /nexsanpool2/charmonium-week-1-20151207.gz already exists and was last modified 2236019 seconds ago
Jan 05 11:22:16 (I) /nexsanpool2/charmonium-week-2-20151214.gz already exists and was last modified 683092 seconds ago
Jan 05 11:22:16 (I) /nexsanpool2/charmonium-week-3-20151221.gz already exists and was last modified 683092 seconds ago
Jan 05 11:22:16 (I) /nexsanpool2/charmonium-week-4-20151228.gz already exists and was last modified 614333 seconds ago
Jan 05 11:22:16 (I) /nexsanpool2/charmonium-week-5-20151231.gz already exists and was last modified 74934 seconds ago
Jan 05 11:22:16 (I) zfs destroy -vr nexsanpool1/charmonium
will destroy nexsanpool1/charmonium@day-5-20151205
will destroy nexsanpool1/charmonium@day-2-20151202
will destroy nexsanpool1/charmonium@day-1-20160101
will destroy nexsanpool1/charmonium@day-25-20151225
will destroy nexsanpool1/charmonium@day-14-20151214
will destroy nexsanpool1/charmonium@day-30-20151230
will destroy nexsanpool1/charmonium@day-13-20151213
will destroy nexsanpool1/charmonium@day-22-20151222
will destroy nexsanpool1/charmonium@day-3-20160103
will destroy nexsanpool1/charmonium@day-29-20151229
will destroy nexsanpool1/charmonium@day-7-20151207
will destroy nexsanpool1/charmonium@day-18-20151218
will destroy nexsanpool1/charmonium@day-4-20160104
will destroy nexsanpool1/charmonium@day-11-20151211
will destroy nexsanpool1/charmonium@day-20-20151220
will destroy nexsanpool1/charmonium@day-27-20151227
will destroy nexsanpool1/charmonium@day-9-20151209
will destroy nexsanpool1/charmonium@day-16-20151216
will destroy nexsanpool1/charmonium@day-10-20151210
will destroy nexsanpool1/charmonium@day-21-20151221
will destroy nexsanpool1/charmonium@day-26-20151226
will destroy nexsanpool1/charmonium@day-8-20151208
will destroy nexsanpool1/charmonium@day-17-20151217
will destroy nexsanpool1/charmonium@day-1-20151201
will destroy nexsanpool1/charmonium@day-2-20160102
will destroy nexsanpool1/charmonium@day-28-20151228
will destroy nexsanpool1/charmonium@day-6-20151206
will destroy nexsanpool1/charmonium@day-19-20151219
will destroy nexsanpool1/charmonium@day-24-20151224
will destroy nexsanpool1/charmonium@day-15-20151215
will destroy nexsanpool1/charmonium@day-31-20151231
will destroy nexsanpool1/charmonium@day-12-20151212
will destroy nexsanpool1/charmonium@day-23-20151223
will destroy nexsanpool1/charmonium@day-4-20151204
will destroy nexsanpool1/charmonium@day-3-20151203
will destroy nexsanpool1/charmonium
Jan 05 11:22:18 (I) zfs send -v projectzfspool/projectzfs/charmonium@day-1-20160101 | zfs recv nexsanpool1/charmonium
send from @ to projectzfspool/projectzfs/charmonium@day-1-20160101 estimated size is 15K
total estimated size is 15K
TIME        SENT   SNAPSHOT
Jan 05 11:22:18 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-1-20151201
will destroy projectzfspool/projectzfs/charmonium@day-1-20151201
will reclaim 0
Jan 05 11:22:19 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-2-20151202
will destroy projectzfspool/projectzfs/charmonium@day-2-20151202
will reclaim 0
Jan 05 11:22:19 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-3-20151203
will destroy projectzfspool/projectzfs/charmonium@day-3-20151203
will reclaim 0
Jan 05 11:22:19 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-4-20151204
will destroy projectzfspool/projectzfs/charmonium@day-4-20151204
will reclaim 0
Jan 05 11:22:19 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-5-20151205
will destroy projectzfspool/projectzfs/charmonium@day-5-20151205
will reclaim 0
Jan 05 11:22:20 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-6-20151206
will destroy projectzfspool/projectzfs/charmonium@day-6-20151206
will reclaim 0
Jan 05 11:22:20 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-7-20151207
will destroy projectzfspool/projectzfs/charmonium@day-7-20151207
will reclaim 0
Jan 05 11:22:20 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-8-20151208
will destroy projectzfspool/projectzfs/charmonium@day-8-20151208
will reclaim 0
Jan 05 11:22:20 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-9-20151209
will destroy projectzfspool/projectzfs/charmonium@day-9-20151209
will reclaim 0
Jan 05 11:22:21 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-10-20151210
will destroy projectzfspool/projectzfs/charmonium@day-10-20151210
will reclaim 0
Jan 05 11:22:21 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-11-20151211
will destroy projectzfspool/projectzfs/charmonium@day-11-20151211
will reclaim 0
Jan 05 11:22:21 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-12-20151212
will destroy projectzfspool/projectzfs/charmonium@day-12-20151212
will reclaim 0
Jan 05 11:22:21 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-13-20151213
will destroy projectzfspool/projectzfs/charmonium@day-13-20151213
will reclaim 0
Jan 05 11:22:22 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-14-20151214
will destroy projectzfspool/projectzfs/charmonium@day-14-20151214
will reclaim 0
Jan 05 11:22:22 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-15-20151215
will destroy projectzfspool/projectzfs/charmonium@day-15-20151215
will reclaim 0
Jan 05 11:22:22 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-16-20151216
will destroy projectzfspool/projectzfs/charmonium@day-16-20151216
will reclaim 0
Jan 05 11:22:22 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-17-20151217
will destroy projectzfspool/projectzfs/charmonium@day-17-20151217
will reclaim 0
Jan 05 11:22:23 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-18-20151218
will destroy projectzfspool/projectzfs/charmonium@day-18-20151218
will reclaim 0
Jan 05 11:22:23 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-19-20151219
will destroy projectzfspool/projectzfs/charmonium@day-19-20151219
will reclaim 0
Jan 05 11:22:23 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-20-20151220
will destroy projectzfspool/projectzfs/charmonium@day-20-20151220
will reclaim 0
Jan 05 11:22:23 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-21-20151221
will destroy projectzfspool/projectzfs/charmonium@day-21-20151221
will reclaim 0
Jan 05 11:22:24 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-22-20151222
will destroy projectzfspool/projectzfs/charmonium@day-22-20151222
will reclaim 0
Jan 05 11:22:24 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-23-20151223
will destroy projectzfspool/projectzfs/charmonium@day-23-20151223
will reclaim 0
Jan 05 11:22:24 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-24-20151224
will destroy projectzfspool/projectzfs/charmonium@day-24-20151224
will reclaim 0
Jan 05 11:22:24 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-25-20151225
will destroy projectzfspool/projectzfs/charmonium@day-25-20151225
will reclaim 0
Jan 05 11:22:25 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-26-20151226
will destroy projectzfspool/projectzfs/charmonium@day-26-20151226
will reclaim 0
Jan 05 11:22:25 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-27-20151227
will destroy projectzfspool/projectzfs/charmonium@day-27-20151227
will reclaim 0
Jan 05 11:22:25 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-28-20151228
will destroy projectzfspool/projectzfs/charmonium@day-28-20151228
will reclaim 0
Jan 05 11:22:25 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-29-20151229
will destroy projectzfspool/projectzfs/charmonium@day-29-20151229
will reclaim 0
Jan 05 11:22:26 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-30-20151230
will destroy projectzfspool/projectzfs/charmonium@day-30-20151230
will reclaim 0
Jan 05 11:22:26 (I) zfs destroy -v projectzfspool/projectzfs/charmonium@day-31-20151231
will destroy projectzfspool/projectzfs/charmonium@day-31-20151231
will reclaim 0
Jan 05 11:22:26 (I) zfs destroy -vr nexsanpool2/charmonium
will destroy nexsanpool2/charmonium@day-1-20151201
will destroy nexsanpool2/charmonium@week-1-20151207
will destroy nexsanpool2/charmonium@week-4-20151228
will destroy nexsanpool2/charmonium@week-2-20151214
will destroy nexsanpool2/charmonium@week-5-20151231
will destroy nexsanpool2/charmonium@week-3-20151221
will destroy nexsanpool2/charmonium
Jan 05 11:22:27 (I) zfs send -v nexsanpool1/charmonium@day-1-20160101 | zfs recv nexsanpool2/charmonium &
send from @ to nexsanpool1/charmonium@day-1-20160101 estimated size is 15K
total estimated size is 15K
TIME        SENT   SNAPSHOT
Jan 05 11:22:27 (I) Replication nexsanpool1->2 is happening in the background
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep charmonium
nexsanpool1/charmonium@day-1-20160101                                   0      -   272K  -
nexsanpool2/charmonium@day-1-20160101                                   0      -   272K  -
projectzfspool/projectzfs/charmonium@day-1-20160101                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20160102                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20160103                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20160104                     0      -   136K  -
[root@dslustre21 zfs.gba]# 

Now running again is not an issue:

[root@dslustre21 zfs.gba]# echo charmonium | ./first-of-month.pl  
Jan 05 11:24:36 (I) Reading dataset list from stdin ..
Jan 05 11:24:36 (I) Checking snapshots for charmonium
Jan 05 11:24:36 (I) /nexsanpool2/charmonium-day-1-20151201.gz already exists and was last modified 2236716 seconds ago
Jan 05 11:24:36 (I) /nexsanpool2/charmonium-week-1-20151207.gz already exists and was last modified 2236159 seconds ago
Jan 05 11:24:36 (I) /nexsanpool2/charmonium-week-2-20151214.gz already exists and was last modified 683232 seconds ago
Jan 05 11:24:36 (I) /nexsanpool2/charmonium-week-3-20151221.gz already exists and was last modified 683232 seconds ago
Jan 05 11:24:36 (I) /nexsanpool2/charmonium-week-4-20151228.gz already exists and was last modified 614473 seconds ago
Jan 05 11:24:36 (I) /nexsanpool2/charmonium-week-5-20151231.gz already exists and was last modified 75074 seconds ago
Jan 05 11:24:36 (E) NOT initializing for charmonium because missingsnaps=6 && day-1-20160101=1 && day-1-20151201=0
[root@dslustre21 zfs.gba]# echo $?
1
[root@dslustre21 zfs.gba]# 

#26 Updated by Gerard Bernabeu Altayo over 4 years ago

Now recreating snaps for charmonium works flawlessly too:

[root@dslustre21 zfs.gba]# echo charmonium | ./replica-zfs-snapshot.pl
Jan 05 11:27:56 (I) Reading dataset list from stdin ..
Jan 05 11:27:56 (I) Making daily snapshots - Snapshot date tag is 5
Jan 05 11:27:56 (I) Replicating snapshots for charmonium
Jan 05 11:27:56 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/charmonium@day-1-20160101 projectzfspool/projectzfs/charmonium@day-2-20160102 | zfs recv nexsanpool1/charmonium@day-2-20160102
send from @day-1-20160101 to projectzfspool/projectzfs/charmonium@day-2-20160102 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Jan 05 11:27:56 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/charmonium@day-2-20160102 projectzfspool/projectzfs/charmonium@day-3-20160103 | zfs recv nexsanpool1/charmonium@day-3-20160103
send from @day-2-20160102 to projectzfspool/projectzfs/charmonium@day-3-20160103 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Jan 05 11:27:57 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/charmonium@day-3-20160103 projectzfspool/projectzfs/charmonium@day-4-20160104 | zfs recv nexsanpool1/charmonium@day-4-20160104
send from @day-3-20160103 to projectzfspool/projectzfs/charmonium@day-4-20160104 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Jan 05 11:27:57 (I) Making weekly snapshots
Jan 05 11:27:57 (I) Checking weekly snapshots for charmonium
There should exist
 day-1-20160101
Will create

Jan 05 11:27:57 (I) Making GZIPs
Jan 05 11:27:57 (I) Making GZIP for charmonium
Jan 05 11:27:57 (I) Creating /nexsanpool2/charmonium-day-1-20160101.gz with: zfs send nexsanpool2/charmonium@day-1-20160101 | gzip > /nexsanpool2/charmonium-day-1-20160101.gz &
[root@dslustre21 zfs.gba]# echo $?
0
[root@dslustre21 zfs.gba]# echo charmonium | ./replica-zfs-snapshot.pl
Jan 05 11:28:05 (I) Reading dataset list from stdin ..
Jan 05 11:28:05 (I) Making daily snapshots - Snapshot date tag is 5
Jan 05 11:28:05 (I) Replicating snapshots for charmonium
Jan 05 11:28:05 (I) Making weekly snapshots
Jan 05 11:28:05 (I) Checking weekly snapshots for charmonium
There should exist
 day-1-20160101
Will create

Jan 05 11:28:05 (I) Making GZIPs
Jan 05 11:28:05 (I) Making GZIP for charmonium
Jan 05 11:28:05 (I) /nexsanpool2/charmonium-day-1-20160101.gz already exists
[root@dslustre21 zfs.gba]# echo $?
0
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep charmonium
nexsanpool1/charmonium@day-1-20160101                               16.0K      -   272K  -
nexsanpool1/charmonium@day-2-20160102                               16.0K      -   272K  -
nexsanpool1/charmonium@day-3-20160103                               16.0K      -   272K  -
nexsanpool1/charmonium@day-4-20160104                                   0      -   272K  -
nexsanpool2/charmonium@day-1-20160101                                   0      -   272K  -
projectzfspool/projectzfs/charmonium@day-1-20160101                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20160102                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20160103                     0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20160104                     0      -   136K  -
[root@dslustre21 zfs.gba]# 

Now doing one experiment that actually has some data:

[root@dslustre21 zfs.gba]# echo c51 | ./first-of-month.pl  &&  zfs list -t snapshot | grep c51 && echo c51 | ./replica-zfs-snapshot.pl; echo $?;  zfs list -t snapshot | grep c51
Jan 05 11:33:38 (I) Reading dataset list from stdin ..
Jan 05 11:33:38 (I) Checking snapshots for c51
Jan 05 11:33:38 (I) /nexsanpool2/c51-day-1-20151201.gz already exists and was last modified 2237193 seconds ago
Jan 05 11:33:38 (I) /nexsanpool2/c51-week-1-20151207.gz already exists and was last modified 2236701 seconds ago
Jan 05 11:33:38 (I) /nexsanpool2/c51-week-2-20151214.gz already exists and was last modified 683774 seconds ago
Jan 05 11:33:38 (I) /nexsanpool2/c51-week-3-20151221.gz already exists and was last modified 683774 seconds ago
Jan 05 11:33:38 (I) /nexsanpool2/c51-week-4-20151228.gz already exists and was last modified 615015 seconds ago
Jan 05 11:33:38 (I) /nexsanpool2/c51-week-5-20151231.gz already exists and was last modified 76099 seconds ago
Jan 05 11:33:38 (I) zfs destroy -vr nexsanpool1/c51
will destroy nexsanpool1/c51@day-3-20160103
will destroy nexsanpool1/c51@day-30-20151230
will destroy nexsanpool1/c51@day-13-20151213
will destroy nexsanpool1/c51@day-22-20151222
will destroy nexsanpool1/c51@day-4-20160104
will destroy nexsanpool1/c51@day-25-20151225
will destroy nexsanpool1/c51@day-14-20151214
will destroy nexsanpool1/c51@day-7-20151207
will destroy nexsanpool1/c51@day-9-20151209
will destroy nexsanpool1/c51@day-27-20151227
will destroy nexsanpool1/c51@day-5-20151205
will destroy nexsanpool1/c51@day-16-20151216
will destroy nexsanpool1/c51@day-1-20160101
will destroy nexsanpool1/c51@day-11-20151211
will destroy nexsanpool1/c51@day-2-20151202
will destroy nexsanpool1/c51@day-20-20151220
will destroy nexsanpool1/c51@day-29-20151229
will destroy nexsanpool1/c51@day-18-20151218
will destroy nexsanpool1/c51@day-28-20151228
will destroy nexsanpool1/c51@day-19-20151219
will destroy nexsanpool1/c51@day-26-20151226
will destroy nexsanpool1/c51@day-4-20151204
will destroy nexsanpool1/c51@day-17-20151217
will destroy nexsanpool1/c51@day-10-20151210
will destroy nexsanpool1/c51@day-3-20151203
will destroy nexsanpool1/c51@day-21-20151221
will destroy nexsanpool1/c51@day-8-20151208
will destroy nexsanpool1/c51@day-2-20160102
will destroy nexsanpool1/c51@day-31-20151231
will destroy nexsanpool1/c51@day-1-20151201
will destroy nexsanpool1/c51@day-12-20151212
will destroy nexsanpool1/c51@day-23-20151223
will destroy nexsanpool1/c51@day-24-20151224
will destroy nexsanpool1/c51@day-15-20151215
will destroy nexsanpool1/c51@day-6-20151206
will destroy nexsanpool1/c51
Jan 05 11:33:40 (I) zfs send -v projectzfspool/projectzfs/c51@day-1-20160101 | zfs recv nexsanpool1/c51
send from @ to projectzfspool/projectzfs/c51@day-1-20160101 estimated size is 990M
total estimated size is 990M
TIME        SENT   SNAPSHOT
11:33:41   54.2K   projectzfspool/projectzfs/c51@day-1-20160101
11:33:42   54.2K   projectzfspool/projectzfs/c51@day-1-20160101
11:33:43   87.1M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:44    196M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:45    295M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:46    397M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:47    497M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:48    604M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:49    711M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:50    825M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:51    896M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:52    921M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:53    950M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:54    975M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:55    994M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:56   1021M   projectzfspool/projectzfs/c51@day-1-20160101
11:33:57   1.03G   projectzfspool/projectzfs/c51@day-1-20160101
11:33:58   1.08G   projectzfspool/projectzfs/c51@day-1-20160101
11:33:59   1.11G   projectzfspool/projectzfs/c51@day-1-20160101
11:34:00   1.13G   projectzfspool/projectzfs/c51@day-1-20160101
11:34:01   1.16G   projectzfspool/projectzfs/c51@day-1-20160101
Jan 05 11:34:03 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-1-20151201
will destroy projectzfspool/projectzfs/c51@day-1-20151201
will reclaim 0
Jan 05 11:34:03 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-2-20151202
will destroy projectzfspool/projectzfs/c51@day-2-20151202
will reclaim 0
Jan 05 11:34:03 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-3-20151203
will destroy projectzfspool/projectzfs/c51@day-3-20151203
will reclaim 0
Jan 05 11:34:07 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-4-20151204
will destroy projectzfspool/projectzfs/c51@day-4-20151204
will reclaim 0
Jan 05 11:34:07 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-5-20151205
will destroy projectzfspool/projectzfs/c51@day-5-20151205
will reclaim 0
Jan 05 11:34:08 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-6-20151206
will destroy projectzfspool/projectzfs/c51@day-6-20151206
will reclaim 0
Jan 05 11:34:08 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-7-20151207
will destroy projectzfspool/projectzfs/c51@day-7-20151207
will reclaim 0
Jan 05 11:34:08 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-8-20151208
will destroy projectzfspool/projectzfs/c51@day-8-20151208
will reclaim 0
Jan 05 11:34:08 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-9-20151209
will destroy projectzfspool/projectzfs/c51@day-9-20151209
will reclaim 0
Jan 05 11:34:09 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-10-20151210
will destroy projectzfspool/projectzfs/c51@day-10-20151210
will reclaim 0
Jan 05 11:34:09 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-11-20151211
will destroy projectzfspool/projectzfs/c51@day-11-20151211
will reclaim 0
Jan 05 11:34:09 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-12-20151212
will destroy projectzfspool/projectzfs/c51@day-12-20151212
will reclaim 0
Jan 05 11:34:09 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-13-20151213
will destroy projectzfspool/projectzfs/c51@day-13-20151213
will reclaim 0
Jan 05 11:34:10 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-14-20151214
will destroy projectzfspool/projectzfs/c51@day-14-20151214
will reclaim 0
Jan 05 11:34:10 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-15-20151215
will destroy projectzfspool/projectzfs/c51@day-15-20151215
will reclaim 0
Jan 05 11:34:10 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-16-20151216
will destroy projectzfspool/projectzfs/c51@day-16-20151216
will reclaim 0
Jan 05 11:34:10 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-17-20151217
will destroy projectzfspool/projectzfs/c51@day-17-20151217
will reclaim 0
Jan 05 11:34:11 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-18-20151218
will destroy projectzfspool/projectzfs/c51@day-18-20151218
will reclaim 0
Jan 05 11:34:11 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-19-20151219
will destroy projectzfspool/projectzfs/c51@day-19-20151219
will reclaim 0
Jan 05 11:34:11 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-20-20151220
will destroy projectzfspool/projectzfs/c51@day-20-20151220
will reclaim 0
Jan 05 11:34:11 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-21-20151221
will destroy projectzfspool/projectzfs/c51@day-21-20151221
will reclaim 0
Jan 05 11:34:12 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-22-20151222
will destroy projectzfspool/projectzfs/c51@day-22-20151222
will reclaim 0
Jan 05 11:34:12 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-23-20151223
will destroy projectzfspool/projectzfs/c51@day-23-20151223
will reclaim 0
Jan 05 11:34:12 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-24-20151224
will destroy projectzfspool/projectzfs/c51@day-24-20151224
will reclaim 0
Jan 05 11:34:13 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-25-20151225
will destroy projectzfspool/projectzfs/c51@day-25-20151225
will reclaim 0
Jan 05 11:34:13 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-26-20151226
will destroy projectzfspool/projectzfs/c51@day-26-20151226
will reclaim 0
Jan 05 11:34:13 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-27-20151227
will destroy projectzfspool/projectzfs/c51@day-27-20151227
will reclaim 0
Jan 05 11:34:13 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-28-20151228
will destroy projectzfspool/projectzfs/c51@day-28-20151228
will reclaim 0
Jan 05 11:34:14 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-29-20151229
will destroy projectzfspool/projectzfs/c51@day-29-20151229
will reclaim 0
Jan 05 11:34:14 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-30-20151230
will destroy projectzfspool/projectzfs/c51@day-30-20151230
will reclaim 0
Jan 05 11:34:14 (I) zfs destroy -v projectzfspool/projectzfs/c51@day-31-20151231
will destroy projectzfspool/projectzfs/c51@day-31-20151231
will reclaim 0
Jan 05 11:34:14 (I) zfs destroy -vr nexsanpool2/c51
will destroy nexsanpool2/c51@week-4-20151228
will destroy nexsanpool2/c51@day-1-20151201
will destroy nexsanpool2/c51@week-2-20151214
will destroy nexsanpool2/c51@week-5-20151231
will destroy nexsanpool2/c51@week-3-20151221
will destroy nexsanpool2/c51@week-1-20151207
will destroy nexsanpool2/c51
Jan 05 11:34:16 (I) zfs send -v nexsanpool1/c51@day-1-20160101 | zfs recv nexsanpool2/c51 &
send from @ to nexsanpool1/c51@day-1-20160101 estimated size is 990M
total estimated size is 990M
TIME        SENT   SNAPSHOT
Jan 05 11:34:16 (I) Replication nexsanpool1->2 is happening in the background
nexsanpool1/c51@day-1-20160101                                          0      -   922M  -
projectzfspool/projectzfs/c51@day-1-20160101                            0      -   747M  -
projectzfspool/projectzfs/c51@day-2-20160102                            0      -   747M  -
projectzfspool/projectzfs/c51@day-3-20160103                            0      -   747M  -
projectzfspool/projectzfs/c51@day-4-20160104                            0      -   747M  -
Jan 05 11:34:16 (I) Reading dataset list from stdin ..
Jan 05 11:34:16 (I) Making daily snapshots - Snapshot date tag is 5
Jan 05 11:34:16 (I) Replicating snapshots for c51
Jan 05 11:34:16 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/c51@day-1-20160101 projectzfspool/projectzfs/c51@day-2-20160102 | zfs recv nexsanpool1/c51@day-2-20160102
send from @day-1-20160101 to projectzfspool/projectzfs/c51@day-2-20160102 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
11:34:17   54.2K   nexsanpool1/c51@day-1-20160101
Jan 05 11:34:17 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/c51@day-2-20160102 projectzfspool/projectzfs/c51@day-3-20160103 | zfs recv nexsanpool1/c51@day-3-20160103
send from @day-2-20160102 to projectzfspool/projectzfs/c51@day-3-20160103 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Jan 05 11:34:17 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/c51@day-3-20160103 projectzfspool/projectzfs/c51@day-4-20160104 | zfs recv nexsanpool1/c51@day-4-20160104
send from @day-3-20160103 to projectzfspool/projectzfs/c51@day-4-20160104 estimated size is 0
total estimated size is 0
TIME        SENT   SNAPSHOT
Jan 05 11:34:18 (I) Making weekly snapshots
Jan 05 11:34:18 (I) Checking weekly snapshots for c51
11:34:18   54.2K   nexsanpool1/c51@day-1-20160101
There should exist
 day-1-20160101
Will create
 day-1-20160101
Jan 05 11:34:18 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/c51@day--7-201601-7 nexsanpool1/c51@day-0-20160100 | zfs recv nexsanpool2/c51@day-1-20160101
WARNING: could not send nexsanpool1/c51@day-0-20160100: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Jan 05 11:34:18 (E) Failed (256) to execute: 

Jan 05 11:34:18 (I) Making GZIPs
Jan 05 11:34:18 (I) Making GZIP for c51
1
nexsanpool1/c51@day-1-20160101                                      16.0K      -   922M  -
nexsanpool1/c51@day-2-20160102                                      16.0K      -   922M  -
nexsanpool1/c51@day-3-20160103                                      16.0K      -   922M  -
nexsanpool1/c51@day-4-20160104                                          0      -   922M  -
projectzfspool/projectzfs/c51@day-1-20160101                            0      -   747M  -
projectzfspool/projectzfs/c51@day-2-20160102                            0      -   747M  -
projectzfspool/projectzfs/c51@day-3-20160103                            0      -   747M  -
projectzfspool/projectzfs/c51@day-4-20160104                            0      -   747M  -
[root@dslustre21 zfs.gba]# 11:34:19    124M   nexsanpool1/c51@day-1-20160101
11:34:20    304M   nexsanpool1/c51@day-1-20160101
11:34:21    504M   nexsanpool1/c51@day-1-20160101
11:34:22    717M   nexsanpool1/c51@day-1-20160101
11:34:23    909M   nexsanpool1/c51@day-1-20160101
11:34:24    987M   nexsanpool1/c51@day-1-20160101
11:34:25   1.06G   nexsanpool1/c51@day-1-20160101
11:34:26   1.13G   nexsanpool1/c51@day-1-20160101
11:34:27   1.16G   nexsanpool1/c51@day-1-20160101

[root@dslustre21 zfs.gba]# echo $?
0

the 1st run will fail to do the GZIP for week1 because the replica to nexsanpool2 is on the background, but a later run of replica-zfs-snapshot.pl (which should run daily) will succeed.

I will do axial manually now, because they’ve data for this month and then I’m thinking to add a cron that runs every 3rd of the month doing the ‘month cleanup and initialization’. It only runs if the GZIPs are all done and does not remove the GZIPs. Since copying the gzips from tape is another server’s dutty.
of course for this month I’ll copy&paste whatever I add in the cron to run for this month, in a screen

[root@dslustre21 zfs.gba]# echo axial | ./first-of-month.pl  &&  zfs list -t snapshot | grep axial && echo axial | ./replica-zfs-snapshot.pl; echo $?;  zfs list -t snapshot | grep axial

#27 Updated by Gerard Bernabeu Altayo over 4 years ago

Running for an experiment with data caused some errors that required manual intervention (as suggested by the script):

[root@dslustre21 zfs.gba]# echo axial | ./first-of-month.pl  &&  zfs list -t snapshot | grep axial && echo axial | ./replica-zfs-snapshot.pl; echo $?;  zfs list -t snapshot | grep axial
Jan 05 12:01:47 (I) Reading dataset list from stdin ..
Jan 05 12:01:47 (I) Checking snapshots for axial
Jan 05 12:01:47 (I) /nexsanpool2/axial-day-1-20151201.gz already exists and was last modified 2237470 seconds ago
Jan 05 12:01:47 (I) /nexsanpool2/axial-week-1-20151207.gz already exists and was last modified 2238389 seconds ago
Jan 05 12:01:47 (I) /nexsanpool2/axial-week-2-20151214.gz already exists and was last modified 685464 seconds ago
Jan 05 12:01:47 (I) /nexsanpool2/axial-week-3-20151221.gz already exists and was last modified 685464 seconds ago
Jan 05 12:01:47 (I) /nexsanpool2/axial-week-4-20151228.gz already exists and was last modified 616703 seconds ago
Jan 05 12:01:47 (I) /nexsanpool2/axial-week-5-20151231.gz already exists and was last modified 11608 seconds ago
Jan 05 12:01:47 (I) zfs destroy -vr nexsanpool1/axial
will destroy nexsanpool1/axial@day-2-20160102
will destroy nexsanpool1/axial@day-1-20151201
will destroy nexsanpool1/axial@day-6-20151206
will destroy nexsanpool1/axial@day-15-20151215
will destroy nexsanpool1/axial@day-24-20151224
will destroy nexsanpool1/axial@day-23-20151223
will destroy nexsanpool1/axial@day-12-20151212
will destroy nexsanpool1/axial@day-8-20151208
will destroy nexsanpool1/axial@day-31-20151231
will destroy nexsanpool1/axial@day-4-20151204
will destroy nexsanpool1/axial@day-19-20151219
will destroy nexsanpool1/axial@day-3-20151203
will destroy nexsanpool1/axial@day-28-20151228
will destroy nexsanpool1/axial@day-21-20151221
will destroy nexsanpool1/axial@day-10-20151210
will destroy nexsanpool1/axial@day-17-20151217
will destroy nexsanpool1/axial@day-26-20151226
will destroy nexsanpool1/axial@day-20-20151220
will destroy nexsanpool1/axial@day-11-20151211
will destroy nexsanpool1/axial@day-16-20151216
will destroy nexsanpool1/axial@day-27-20151227
will destroy nexsanpool1/axial@day-5-20151205
will destroy nexsanpool1/axial@day-1-20160101
will destroy nexsanpool1/axial@day-18-20151218
will destroy nexsanpool1/axial@day-2-20151202
will destroy nexsanpool1/axial@day-29-20151229
will destroy nexsanpool1/axial@day-14-20151214
will destroy nexsanpool1/axial@day-25-20151225
will destroy nexsanpool1/axial@day-22-20151222
will destroy nexsanpool1/axial@day-13-20151213
will destroy nexsanpool1/axial@day-9-20151209
will destroy nexsanpool1/axial@day-30-20151230
will destroy nexsanpool1/axial@day-3-20160103
will destroy nexsanpool1/axial@day-4-20160104
will destroy nexsanpool1/axial@day-7-20151207
will destroy nexsanpool1/axial
Jan 05 12:01:49 (I) zfs send -v projectzfspool/projectzfs/axial@day-1-20160101 | zfs recv nexsanpool1/axial
send from @ to projectzfspool/projectzfs/axial@day-1-20160101 estimated size is 14.4G
total estimated size is 14.4G
TIME        SENT   SNAPSHOT
12:01:50   47.6K   projectzfspool/projectzfs/axial@day-1-20160101
12:01:51   47.6K   projectzfspool/projectzfs/axial@day-1-20160101
12:01:52   75.5M   projectzfspool/projectzfs/axial@day-1-20160101
12:01:53    321M   projectzfspool/projectzfs/axial@day-1-20160101
12:01:54    590M   projectzfspool/projectzfs/axial@day-1-20160101
12:01:55    789M   projectzfspool/projectzfs/axial@day-1-20160101
12:01:56    931M   projectzfspool/projectzfs/axial@day-1-20160101
12:01:57    968M   projectzfspool/projectzfs/axial@day-1-20160101
12:01:58   1.02G   projectzfspool/projectzfs/axial@day-1-20160101
12:01:59   1.05G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:00   1.09G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:01   1.12G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:02   1.15G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:03   1.17G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:04   1.19G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:05   1.21G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:06   1.22G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:07   1.24G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:08   1.28G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:09   1.31G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:10   1.33G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:11   1.35G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:12   1.38G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:13   1.40G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:14   1.44G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:15   1.51G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:16   1.53G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:17   1.55G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:18   1.58G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:19   1.60G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:20   1.63G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:21   1.67G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:22   1.82G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:23   2.00G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:24   2.22G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:25   2.44G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:26   2.68G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:27   2.88G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:28   3.13G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:29   3.36G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:30   3.54G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:31   3.77G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:32   3.99G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:33   4.22G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:34   4.48G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:35   4.65G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:36   4.88G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:37   5.10G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:38   5.34G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:39   5.58G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:40   5.77G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:41   5.98G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:42   6.20G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:43   6.44G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:44   6.68G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:45   6.87G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:46   7.09G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:47   7.28G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:48   7.35G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:49   7.38G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:50   7.40G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:51   7.44G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:52   7.47G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:53   7.49G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:54   7.52G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:55   7.55G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:56   7.56G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:57   7.56G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:58   7.57G   projectzfspool/projectzfs/axial@day-1-20160101
12:02:59   7.61G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:00   7.72G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:01   7.75G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:02   7.76G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:03   7.80G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:04   7.87G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:05   7.99G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:06   8.06G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:07   8.11G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:08   8.20G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:09   8.40G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:10   8.65G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:11   8.88G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:12   9.11G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:13   9.34G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:14   9.56G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:15   9.80G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:16   10.0G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:17   10.2G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:18   10.4G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:19   10.7G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:20   10.9G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:21   11.0G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:22   11.0G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:23   11.0G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:24   11.1G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:25   11.3G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:26   11.4G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:27   11.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:28   11.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:29   11.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:30   11.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:31   11.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:32   11.6G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:33   11.7G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:34   11.7G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:35   11.9G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:36   12.1G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:37   12.3G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:38   12.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:39   12.7G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:40   12.8G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:41   13.0G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:42   13.2G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:43   13.3G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:44   13.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:45   13.6G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:46   13.7G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:47   13.9G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:48   14.0G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:49   14.1G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:50   14.2G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:51   14.4G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:52   14.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:53   14.5G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:54   14.6G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:55   14.6G   projectzfspool/projectzfs/axial@day-1-20160101
12:03:56   14.6G   projectzfspool/projectzfs/axial@day-1-20160101
Jan 05 12:03:59 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-1-20151201
will destroy projectzfspool/projectzfs/axial@day-1-20151201
will reclaim 0
Jan 05 12:03:59 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-2-20151202
will destroy projectzfspool/projectzfs/axial@day-2-20151202
will reclaim 3.64M
Jan 05 12:03:59 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-3-20151203
will destroy projectzfspool/projectzfs/axial@day-3-20151203
will reclaim 0
Jan 05 12:04:00 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-4-20151204
will destroy projectzfspool/projectzfs/axial@day-4-20151204
will reclaim 0
Jan 05 12:04:00 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-5-20151205
will destroy projectzfspool/projectzfs/axial@day-5-20151205
will reclaim 0
Jan 05 12:04:00 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-6-20151206
will destroy projectzfspool/projectzfs/axial@day-6-20151206
will reclaim 0
Jan 05 12:04:00 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-7-20151207
will destroy projectzfspool/projectzfs/axial@day-7-20151207
will reclaim 0
Jan 05 12:04:01 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-8-20151208
will destroy projectzfspool/projectzfs/axial@day-8-20151208
will reclaim 0
Jan 05 12:04:01 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-9-20151209
will destroy projectzfspool/projectzfs/axial@day-9-20151209
will reclaim 0
Jan 05 12:04:01 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-10-20151210
will destroy projectzfspool/projectzfs/axial@day-10-20151210
will reclaim 0
Jan 05 12:04:01 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-11-20151211
will destroy projectzfspool/projectzfs/axial@day-11-20151211
will reclaim 0
Jan 05 12:04:02 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-12-20151212
will destroy projectzfspool/projectzfs/axial@day-12-20151212
will reclaim 0
Jan 05 12:04:02 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-13-20151213
will destroy projectzfspool/projectzfs/axial@day-13-20151213
will reclaim 0
Jan 05 12:04:02 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-14-20151214
will destroy projectzfspool/projectzfs/axial@day-14-20151214
will reclaim 0
Jan 05 12:04:02 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-15-20151215
will destroy projectzfspool/projectzfs/axial@day-15-20151215
will reclaim 0
Jan 05 12:04:03 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-16-20151216
will destroy projectzfspool/projectzfs/axial@day-16-20151216
will reclaim 0
Jan 05 12:04:04 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-17-20151217
will destroy projectzfspool/projectzfs/axial@day-17-20151217
will reclaim 0
Jan 05 12:04:07 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-18-20151218
will destroy projectzfspool/projectzfs/axial@day-18-20151218
will reclaim 0
Jan 05 12:04:07 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-19-20151219
will destroy projectzfspool/projectzfs/axial@day-19-20151219
will reclaim 0
Jan 05 12:04:07 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-20-20151220
will destroy projectzfspool/projectzfs/axial@day-20-20151220
will reclaim 0
Jan 05 12:04:08 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-21-20151221
will destroy projectzfspool/projectzfs/axial@day-21-20151221
will reclaim 0
Jan 05 12:04:08 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-22-20151222
will destroy projectzfspool/projectzfs/axial@day-22-20151222
will reclaim 0
Jan 05 12:04:08 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-23-20151223
will destroy projectzfspool/projectzfs/axial@day-23-20151223
will reclaim 0
Jan 05 12:04:08 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-24-20151224
will destroy projectzfspool/projectzfs/axial@day-24-20151224
will reclaim 0
Jan 05 12:04:09 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-25-20151225
will destroy projectzfspool/projectzfs/axial@day-25-20151225
will reclaim 0
Jan 05 12:04:09 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-26-20151226
will destroy projectzfspool/projectzfs/axial@day-26-20151226
will reclaim 5.94M
Jan 05 12:04:09 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-27-20151227
will destroy projectzfspool/projectzfs/axial@day-27-20151227
will reclaim 4.45M
Jan 05 12:04:10 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-28-20151228
will destroy projectzfspool/projectzfs/axial@day-28-20151228
will reclaim 4.86M
Jan 05 12:04:10 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-29-20151229
will destroy projectzfspool/projectzfs/axial@day-29-20151229
will reclaim 536K
Jan 05 12:04:10 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-30-20151230
will destroy projectzfspool/projectzfs/axial@day-30-20151230
will reclaim 5.75M
Jan 05 12:04:11 (I) zfs destroy -v projectzfspool/projectzfs/axial@day-31-20151231
will destroy projectzfspool/projectzfs/axial@day-31-20151231
will reclaim 3.65M
Jan 05 12:04:11 (I) zfs destroy -vr nexsanpool2/axial
will destroy nexsanpool2/axial@week-1-20151207
will destroy nexsanpool2/axial@week-3-20151221
will destroy nexsanpool2/axial@day-1-20151201
will destroy nexsanpool2/axial@week-5-20151231
will destroy nexsanpool2/axial@week-4-20151228
will destroy nexsanpool2/axial@week-2-20151214
will destroy nexsanpool2/axial
Jan 05 12:04:13 (I) zfs send -v nexsanpool1/axial@day-1-20160101 | zfs recv nexsanpool2/axial &
send from @ to nexsanpool1/axial@day-1-20160101 estimated size is 14.4G
total estimated size is 14.4G
TIME        SENT   SNAPSHOT
Jan 05 12:04:13 (I) Replication nexsanpool1->2 is happening in the background
nexsanpool1/axial@day-1-20160101                                     296K      -  12.1G  -
projectzfspool/projectzfs/axial@day-1-20160101                      5.66M      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20160102                      3.62M      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20160103                       792K      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20160104                       148K      -  12.3G  -
Jan 05 12:04:13 (I) Reading dataset list from stdin ..
Jan 05 12:04:13 (I) Making daily snapshots - Snapshot date tag is 5
Jan 05 12:04:13 (I) Replicating snapshots for axial
Jan 05 12:04:13 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/axial@day-1-20160101 projectzfspool/projectzfs/axial@day-2-20160102 | zfs recv nexsanpool1/axial@day-2-20160102
send from @day-1-20160101 to projectzfspool/projectzfs/axial@day-2-20160102 estimated size is 11.0M
total estimated size is 11.0M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: destination nexsanpool1/axial has been modified
since most recent snapshot
warning: cannot send 'projectzfspool/projectzfs/axial@day-2-20160102': Broken pipe
Jan 05 12:04:13 (E) Failed (256) to execute: zfs send -v -i projectzfspool/projectzfs/axial@day-1-20160101 projectzfspool/projectzfs/axial@day-2-20160102 | zfs recv nexsanpool1/axial@day-2-20160102 
    To manually fix try using 'zfs recv -F'

Jan 05 12:04:13 (I) Making weekly snapshots
Jan 05 12:04:13 (I) Checking weekly snapshots for axial
cannot open 'nexsanpool2/axial': dataset does not exist
There should exist
 day-1-20160101
Will create
 day-1-20160101
Jan 05 12:04:14 (I) Creating weekly snapshot: zfs send -v -i nexsanpool1/axial@day--7-201601-7 nexsanpool1/axial@day-0-20160100 | zfs recv nexsanpool2/axial@day-1-20160101
WARNING: could not send nexsanpool1/axial@day-0-20160100: does not exist
total estimated size is 0
cannot receive: failed to read from stream
Jan 05 12:04:14 (E) Failed (256) to execute: zfs send -v -i nexsanpool1/axial@day--7-201601-7 nexsanpool1/axial@day-0-20160100 | zfs recv nexsanpool2/axial@day-1-20160101

Jan 05 12:04:14 (I) Making GZIPs
Jan 05 12:04:14 (I) Making GZIP for axial
cannot open 'nexsanpool2/axial': dataset does not exist
1
12:04:14   47.6K   nexsanpool1/axial@day-1-20160101
nexsanpool1/axial@day-1-20160101                                     296K      -  12.1G  -
projectzfspool/projectzfs/axial@day-1-20160101                      5.66M      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20160102                      3.62M      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20160103                       792K      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20160104                       148K      -  12.3G  -
[root@dslustre21 zfs.gba]# 12:04:15   47.6K   nexsanpool1/axial@day-1-20160101
12:04:16   49.7M   nexsanpool1/axial@day-1-20160101
12:04:17    194M   nexsanpool1/axial@day-1-20160101
12:04:18    364M   nexsanpool1/axial@day-1-20160101
12:04:19    539M   nexsanpool1/axial@day-1-20160101
12:04:20    694M   nexsanpool1/axial@day-1-20160101
12:04:21    844M   nexsanpool1/axial@day-1-20160101
12:04:22    949M   nexsanpool1/axial@day-1-20160101
12:04:23    980M   nexsanpool1/axial@day-1-20160101
12:04:24   1.03G   nexsanpool1/axial@day-1-20160101
12:04:25   1.05G   nexsanpool1/axial@day-1-20160101
12:04:26   1.11G   nexsanpool1/axial@day-1-20160101
12:04:27   1.13G   nexsanpool1/axial@day-1-20160101
12:04:28   1.16G   nexsanpool1/axial@day-1-20160101
12:04:29   1.18G   nexsanpool1/axial@day-1-20160101
12:04:30   1.21G   nexsanpool1/axial@day-1-20160101
12:04:31   1.24G   nexsanpool1/axial@day-1-20160101
12:04:32   1.27G   nexsanpool1/axial@day-1-20160101
12:04:33   1.29G   nexsanpool1/axial@day-1-20160101
12:04:34   1.32G   nexsanpool1/axial@day-1-20160101
12:04:35   1.35G   nexsanpool1/axial@day-1-20160101
12:04:36   1.37G   nexsanpool1/axial@day-1-20160101
12:04:37   1.40G   nexsanpool1/axial@day-1-20160101
12:04:38   1.42G   nexsanpool1/axial@day-1-20160101
12:04:39   1.52G   nexsanpool1/axial@day-1-20160101
12:04:40   1.54G   nexsanpool1/axial@day-1-20160101
12:04:41   1.57G   nexsanpool1/axial@day-1-20160101
12:04:42   1.58G   nexsanpool1/axial@day-1-20160101
12:04:43   1.62G   nexsanpool1/axial@day-1-20160101
12:04:44   1.64G   nexsanpool1/axial@day-1-20160101
12:04:45   1.77G   nexsanpool1/axial@day-1-20160101
12:04:46   1.96G   nexsanpool1/axial@day-1-20160101
12:04:47   2.19G   nexsanpool1/axial@day-1-20160101
12:04:48   2.22G   nexsanpool1/axial@day-1-20160101
12:04:49   2.47G   nexsanpool1/axial@day-1-20160101
12:04:50   2.73G   nexsanpool1/axial@day-1-20160101
12:04:51   2.77G   nexsanpool1/axial@day-1-20160101
12:04:52   2.88G   nexsanpool1/axial@day-1-20160101
12:04:53   3.05G   nexsanpool1/axial@day-1-20160101
12:04:54   3.25G   nexsanpool1/axial@day-1-20160101
12:04:55   3.35G   nexsanpool1/axial@day-1-20160101
12:04:56   3.44G   nexsanpool1/axial@day-1-20160101
12:04:57   3.56G   nexsanpool1/axial@day-1-20160101
12:04:58   3.66G   nexsanpool1/axial@day-1-20160101
12:04:59   3.75G   nexsanpool1/axial@day-1-20160101
12:05:00   3.83G   nexsanpool1/axial@day-1-20160101
12:05:01   3.91G   nexsanpool1/axial@day-1-20160101
12:05:02   3.94G   nexsanpool1/axial@day-1-20160101
12:05:03   3.94G   nexsanpool1/axial@day-1-20160101
12:05:04   4.05G   nexsanpool1/axial@day-1-20160101
12:05:05   4.18G   nexsanpool1/axial@day-1-20160101
12:05:06   4.31G   nexsanpool1/axial@day-1-20160101
12:05:07   4.42G   nexsanpool1/axial@day-1-20160101
12:05:08   4.52G   nexsanpool1/axial@day-1-20160101
12:05:09   4.52G   nexsanpool1/axial@day-1-20160101
12:05:10   4.58G   nexsanpool1/axial@day-1-20160101
12:05:11   4.77G   nexsanpool1/axial@day-1-20160101
12:05:12   5.00G   nexsanpool1/axial@day-1-20160101
12:05:13   5.10G   nexsanpool1/axial@day-1-20160101
12:05:14   5.16G   nexsanpool1/axial@day-1-20160101
12:05:15   5.32G   nexsanpool1/axial@day-1-20160101
12:05:16   5.47G   nexsanpool1/axial@day-1-20160101
12:05:17   5.58G   nexsanpool1/axial@day-1-20160101
12:05:18   5.68G   nexsanpool1/axial@day-1-20160101
12:05:19   5.68G   nexsanpool1/axial@day-1-20160101
12:05:20   5.77G   nexsanpool1/axial@day-1-20160101
12:05:21   5.96G   nexsanpool1/axial@day-1-20160101
12:05:22   6.13G   nexsanpool1/axial@day-1-20160101
12:05:23   6.26G   nexsanpool1/axial@day-1-20160101
12:05:24   6.26G   nexsanpool1/axial@day-1-20160101
12:05:25   6.43G   nexsanpool1/axial@day-1-20160101
12:05:26   6.61G   nexsanpool1/axial@day-1-20160101
12:05:27   6.74G   nexsanpool1/axial@day-1-20160101
12:05:28   6.90G   nexsanpool1/axial@day-1-20160101
12:05:29   7.13G   nexsanpool1/axial@day-1-20160101
12:05:30   7.25G   nexsanpool1/axial@day-1-20160101
12:05:31   7.35G   nexsanpool1/axial@day-1-20160101
12:05:32   7.38G   nexsanpool1/axial@day-1-20160101
12:05:33   7.42G   nexsanpool1/axial@day-1-20160101
12:05:34   7.46G   nexsanpool1/axial@day-1-20160101
12:05:35   7.49G   nexsanpool1/axial@day-1-20160101
12:05:36   7.52G   nexsanpool1/axial@day-1-20160101
12:05:37   7.56G   nexsanpool1/axial@day-1-20160101
12:05:38   7.69G   nexsanpool1/axial@day-1-20160101
12:05:39   7.75G   nexsanpool1/axial@day-1-20160101
12:05:40   7.78G   nexsanpool1/axial@day-1-20160101
12:05:41   7.95G   nexsanpool1/axial@day-1-20160101
12:05:42   8.15G   nexsanpool1/axial@day-1-20160101
12:05:43   8.38G   nexsanpool1/axial@day-1-20160101
12:05:44   8.56G   nexsanpool1/axial@day-1-20160101
12:05:45   8.76G   nexsanpool1/axial@day-1-20160101
12:05:46   8.89G   nexsanpool1/axial@day-1-20160101
12:05:47   9.04G   nexsanpool1/axial@day-1-20160101
12:05:48   9.17G   nexsanpool1/axial@day-1-20160101
12:05:49   9.35G   nexsanpool1/axial@day-1-20160101
12:05:50   9.52G   nexsanpool1/axial@day-1-20160101
12:05:51   9.77G   nexsanpool1/axial@day-1-20160101
12:05:52   9.92G   nexsanpool1/axial@day-1-20160101
12:05:53   10.1G   nexsanpool1/axial@day-1-20160101
12:05:54   10.2G   nexsanpool1/axial@day-1-20160101
12:05:55   10.4G   nexsanpool1/axial@day-1-20160101
12:05:56   10.6G   nexsanpool1/axial@day-1-20160101
12:05:57   10.8G   nexsanpool1/axial@day-1-20160101
12:05:58   11.0G   nexsanpool1/axial@day-1-20160101
12:05:59   11.1G   nexsanpool1/axial@day-1-20160101
12:06:00   11.3G   nexsanpool1/axial@day-1-20160101
12:06:01   11.5G   nexsanpool1/axial@day-1-20160101
12:06:02   11.5G   nexsanpool1/axial@day-1-20160101
12:06:03   11.6G   nexsanpool1/axial@day-1-20160101
12:06:04   11.7G   nexsanpool1/axial@day-1-20160101
12:06:05   12.0G   nexsanpool1/axial@day-1-20160101
12:06:06   12.2G   nexsanpool1/axial@day-1-20160101
12:06:07   12.3G   nexsanpool1/axial@day-1-20160101
12:06:08   12.5G   nexsanpool1/axial@day-1-20160101
12:06:09   12.7G   nexsanpool1/axial@day-1-20160101
12:06:10   12.8G   nexsanpool1/axial@day-1-20160101
12:06:11   13.0G   nexsanpool1/axial@day-1-20160101
12:06:12   13.2G   nexsanpool1/axial@day-1-20160101
12:06:13   13.4G   nexsanpool1/axial@day-1-20160101
12:06:14   13.6G   nexsanpool1/axial@day-1-20160101
12:06:15   13.7G   nexsanpool1/axial@day-1-20160101
12:06:16   13.9G   nexsanpool1/axial@day-1-20160101
12:06:17   14.1G   nexsanpool1/axial@day-1-20160101
12:06:18   14.2G   nexsanpool1/axial@day-1-20160101
12:06:19   14.4G   nexsanpool1/axial@day-1-20160101
12:06:20   14.6G   nexsanpool1/axial@day-1-20160101

[root@dslustre21 zfs.gba]# echo $?
0
[root@dslustre21 zfs.gba]# 

Now a second run shows the errors more clearly:

[root@dslustre21 zfs.gba]# echo axial | ./first-of-month.pl  &&  zfs list -t snapshot | grep axial && echo axial | ./replica-zfs-snapshot.pl; echo $?;  zfs list -t snapshot | grep axial
Jan 05 12:12:32 (I) Reading dataset list from stdin ..
Jan 05 12:12:32 (I) Checking snapshots for axial
Jan 05 12:12:32 (I) /nexsanpool2/axial-day-1-20151201.gz already exists and was last modified 2238115 seconds ago
Jan 05 12:12:32 (I) /nexsanpool2/axial-week-1-20151207.gz already exists and was last modified 2239034 seconds ago
Jan 05 12:12:32 (I) /nexsanpool2/axial-week-2-20151214.gz already exists and was last modified 686109 seconds ago
Jan 05 12:12:32 (I) /nexsanpool2/axial-week-3-20151221.gz already exists and was last modified 686109 seconds ago
Jan 05 12:12:32 (I) /nexsanpool2/axial-week-4-20151228.gz already exists and was last modified 617348 seconds ago
Jan 05 12:12:32 (I) /nexsanpool2/axial-week-5-20151231.gz already exists and was last modified 12253 seconds ago
Jan 05 12:12:32 (E) NOT initializing for axial because missingsnaps=6 && day-1-20160101=1 && day-1-20151201=0
1
nexsanpool1/axial@day-1-20160101                                     296K      -  12.1G  -
nexsanpool2/axial@day-1-20160101                                     296K      -  12.1G  -
projectzfspool/projectzfs/axial@day-1-20160101                      5.66M      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20160102                      3.62M      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20160103                       792K      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20160104                       148K      -  12.3G  -
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep axial && echo axial | ./replica-zfs-snapshot.pl; echo $?
nexsanpool1/axial@day-1-20160101                                     296K      -  12.1G  -
nexsanpool2/axial@day-1-20160101                                     296K      -  12.1G  -
projectzfspool/projectzfs/axial@day-1-20160101                      5.66M      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20160102                      3.62M      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20160103                       792K      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20160104                       148K      -  12.3G  -
Jan 05 12:12:48 (I) Reading dataset list from stdin ..
Jan 05 12:12:48 (I) Making daily snapshots - Snapshot date tag is 5
Jan 05 12:12:48 (I) Replicating snapshots for axial
Jan 05 12:12:48 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/axial@day-1-20160101 projectzfspool/projectzfs/axial@day-2-20160102 | zfs recv nexsanpool1/axial@day-2-20160102
send from @day-1-20160101 to projectzfspool/projectzfs/axial@day-2-20160102 estimated size is 11.0M
total estimated size is 11.0M
TIME        SENT   SNAPSHOT
cannot receive incremental stream: destination nexsanpool1/axial has been modified
since most recent snapshot
warning: cannot send 'projectzfspool/projectzfs/axial@day-2-20160102': Broken pipe
Jan 05 12:12:48 (E) Failed (256) to execute: zfs send -v -i projectzfspool/projectzfs/axial@day-1-20160101 projectzfspool/projectzfs/axial@day-2-20160102 | zfs recv nexsanpool1/axial@day-2-20160102 
    To manually fix try using 'zfs recv -F'

Jan 05 12:12:48 (I) Making weekly snapshots
Jan 05 12:12:48 (I) Checking weekly snapshots for axial
There should exist
 day-1-20160101
Will create

Jan 05 12:12:48 (I) Making GZIPs
Jan 05 12:12:48 (I) Making GZIP for axial
Jan 05 12:12:48 (I) Creating /nexsanpool2/axial-day-1-20160101.gz with: zfs send nexsanpool2/axial@day-1-20160101 | gzip > /nexsanpool2/axial-day-1-20160101.gz &
1
[root@dslustre21 zfs.gba]# 

So the manual action is:

[root@dslustre21 zfs.gba]# zfs send -v -i projectzfspool/projectzfs/axial@day-1-20160101 projectzfspool/projectzfs/axial@day-2-20160102 | zfs recv -F nexsanpool1/axial@day-2-20160102
send from @day-1-20160101 to projectzfspool/projectzfs/axial@day-2-20160102 estimated size is 11.0M
total estimated size is 11.0M
TIME        SENT   SNAPSHOT
[root@dslustre21 zfs.gba]# 

I'm thinking to run with -F by default...

Subsequent runs succeed:

[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep axial && echo axial | ./replica-zfs-snapshot.pl; echo $?
nexsanpool1/axial@day-1-20160101                                    7.39M      -  12.1G  -
nexsanpool1/axial@day-2-20160102                                        0      -  12.1G  -
nexsanpool2/axial@day-1-20160101                                     296K      -  12.1G  -
projectzfspool/projectzfs/axial@day-1-20160101                      5.66M      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20160102                      3.62M      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20160103                       792K      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20160104                       148K      -  12.3G  -
Jan 05 12:13:45 (I) Reading dataset list from stdin ..
Jan 05 12:13:45 (I) Making daily snapshots - Snapshot date tag is 5
Jan 05 12:13:45 (I) Replicating snapshots for axial
Jan 05 12:13:45 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/axial@day-2-20160102 projectzfspool/projectzfs/axial@day-3-20160103 | zfs recv nexsanpool1/axial@day-3-20160103
send from @day-2-20160102 to projectzfspool/projectzfs/axial@day-3-20160103 estimated size is 7.74M
total estimated size is 7.74M
TIME        SENT   SNAPSHOT
Jan 05 12:13:46 (I) Creating daily snapshot: zfs send -v -i projectzfspool/projectzfs/axial@day-3-20160103 projectzfspool/projectzfs/axial@day-4-20160104 | zfs recv nexsanpool1/axial@day-4-20160104
send from @day-3-20160103 to projectzfspool/projectzfs/axial@day-4-20160104 estimated size is 711M
total estimated size is 711M
TIME        SENT   SNAPSHOT
12:13:47   56.8M   projectzfspool/projectzfs/axial@day-4-20160104
12:13:48    138M   projectzfspool/projectzfs/axial@day-4-20160104
12:13:49    223M   projectzfspool/projectzfs/axial@day-4-20160104
12:13:50    313M   projectzfspool/projectzfs/axial@day-4-20160104
12:13:51    412M   projectzfspool/projectzfs/axial@day-4-20160104
12:13:52    511M   projectzfspool/projectzfs/axial@day-4-20160104
12:13:53    592M   projectzfspool/projectzfs/axial@day-4-20160104
12:13:54    684M   projectzfspool/projectzfs/axial@day-4-20160104
Jan 05 12:13:55 (I) Making weekly snapshots
Jan 05 12:13:55 (I) Checking weekly snapshots for axial
There should exist
 day-1-20160101
Will create

Jan 05 12:13:55 (I) Making GZIPs
Jan 05 12:13:55 (I) Making GZIP for axial
Jan 05 12:13:55 (I) /nexsanpool2/axial-day-1-20160101.gz already exists
0
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# zfs list -t snapshot | grep axial && echo axial | ./replica-zfs-snapshot.pl; echo $?
nexsanpool1/axial@day-1-20160101                                    7.39M      -  12.1G  -
nexsanpool1/axial@day-2-20160102                                    5.00M      -  12.1G  -
nexsanpool1/axial@day-3-20160103                                    1.53M      -  12.1G  -
nexsanpool1/axial@day-4-20160104                                        0      -  12.5G  -
nexsanpool2/axial@day-1-20160101                                     296K      -  12.1G  -
projectzfspool/projectzfs/axial@day-1-20160101                      5.66M      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20160102                      3.62M      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20160103                       792K      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20160104                       148K      -  12.3G  -
Jan 05 12:14:49 (I) Reading dataset list from stdin ..
Jan 05 12:14:49 (I) Making daily snapshots - Snapshot date tag is 5
Jan 05 12:14:49 (I) Replicating snapshots for axial
Jan 05 12:14:49 (I) Making weekly snapshots
Jan 05 12:14:49 (I) Checking weekly snapshots for axial
There should exist
 day-1-20160101
Will create

Jan 05 12:14:49 (I) Making GZIPs
Jan 05 12:14:49 (I) Making GZIP for axial
Jan 05 12:14:49 (I) /nexsanpool2/axial-day-1-20160101.gz already exists
0
[root@dslustre21 zfs.gba]# 

#28 Updated by Gerard Bernabeu Altayo over 4 years ago

I've just moved the scripts to the zfs-scripts directory, adding everything to a local GIT repo:

[root@dslustre21 zfs-scripts]# pwd
/root/zfs-scripts
[root@dslustre21 zfs-scripts]# ls -lah
total 6.2M
drwxr-xr-x   3 root root 4.0K Jan  5 12:29 .
dr-xr-x---. 19 root root 4.0K Jan  5 12:35 ..
-rwxr-xr-x   1 root root 3.1K Nov  2 12:12 daily-zfs-snapshot
-rw-r--r--   1 root root 2.7M Jan  4 23:59 daily-zfs-snapshot.log
-rw-r--r--   1 root root 1.6M Jul 12 23:59 daily-zfs-snapshot.log.savedon07132015
-rw-r--r--   1 root root 497K Aug 13 23:59 daily-zfs-snapshot.log.savedon08142015
-rw-r--r--   1 root root 1.3M Oct 18 23:59 daily-zfs-snapshot.log.savedon10192015
-rwxr-xr-x   1 root root 1.5K Nov  2 12:09 daily-zfs-snapshot.work
-rw-r--r--   1 root root  426 Sep 10 13:15 dataset-list
-rw-r--r--   1 root root  114 Sep 10 13:15 dataset-list-reduced
-rwxr-xr-x   1 root root 3.0K Nov  2 12:16 decision-noemail.pl
-rwxr-xr-x   1 root root 3.0K Nov  2 12:16 decision.pl
-rwxr-xr-x   1 root root  900 Aug 11 09:00 delete-snapshots
-rwxr-xr-x   1 root root 4.6K Jan  5 11:21 first-of-month.pl
-rw-r--r--   1 root root  309 Aug  7 11:24 generate-gzips.txt
drwxr-xr-x   8 root root 4.0K Jan  5 12:37 .git
-rw-------   1 root root    0 Dec  9 12:05 nohup.out
-rwxr-xr-x   1 root root 5.6K Jan  5 12:29 replica-zfs-snapshot.pl
-rwxr-xr-x   1 root root 4.4K Jan  4 11:51 replica-zfs-snapshot.sh.good.only4weeks
-rw-r--r--   1 root root 1.6K Dec  3 09:05 runme.sh
-rwxr-xr-x   1 root root 1.1K May 11  2015 zfs-send-recv
[root@dslustre21 zfs-scripts]# git status
# On branch master
nothing to commit (working directory clean)
[root@dslustre21 zfs-scripts]# 

Now adding the crons:

[root@dslustre21 zfs.gba]# crontab -e

# Daily ZFS snapshot. Run at midnight.
59 23 * * * (cat /root/zfs-scripts/dataset-list | /root/zfs-scripts/daily-zfs-snapshot >> /root/zfs-scripts/daily-zfs-snapshot.log 2>&1)

#Daily ZFS snapshot replication. When needed the script also makes the Weekly and GZIP backups
59 1 * * * (cat /root/zfs-scripts/dataset-list-reduced | /root/zfs-scripts/replica-zfs-snapshot.pl >> /root/zfs-scripts/daily-zfs-replica-snapshot.log 2>&1)

#Monthtly ZFS snapshot cleanup. Run on the 2nd of the month to make sure all GZIPs have been created. Copy to tape and removal is not automated here! Run from [root@lqcdsrm ~]# vim /home/gerard1/backups_to_tape.sh. For now it requires manual cleanup, it needs to be automated
00 4 2 * * (cat /root/zfs-scripts/dataset-list-reduced | /root/zfs-scripts/first-of-month.pl >> /root/zfs-scripts/monthly-zfs-initialization.log 2>&1)

And running the initialization manually, from a screen:

[root@dslustre21 zfs.gba]# ls -lah /nexsanpool2/*2016*
-rw-r--r-- 1 root root 8.1G Jan  5 12:32 /nexsanpool2/axial-day-1-20160101.gz
-rw-r--r-- 1 root root 563M Jan  5 11:45 /nexsanpool2/c51-day-1-20160101.gz
-rw-r--r-- 1 root root 2.7K Jan  5 11:27 /nexsanpool2/charmonium-day-1-20160101.gz
[root@dslustre21 zfs.gba]# date
Tue Jan  5 12:45:43 CST 2016
[root@dslustre21 zfs.gba]# screen

[root@dslustre21 zfs.gba]# cat /root/zfs-scripts/dataset-list-reduced | /root/zfs-scripts/first-of-month.pl >> /root/zfs-scripts/monthly-zfs-initialization.log 2>&1
[root@dslustre21 zfs.gba]# 
[root@dslustre21 zfs.gba]# screen
[detached]

Will let it run, tomorrow's cron run should succeed to replicate all snapshots day 2-5.

#29 Updated by Gerard Bernabeu Altayo over 4 years ago

There are no errors in the log file:

[root@dslustre21 ~]# grep '(E)' /root/zfs-scripts/monthly-zfs-initialization.log
Jan 05 12:48:34 (E) NOT initializing for axial because missingsnaps=6 && day-1-20160101=1 && day-1-20151201=0
Jan 05 12:48:34 (E) NOT initializing for charmonium because missingsnaps=6 && day-1-20160101=1 && day-1-20151201=0
Jan 05 12:48:34 (E) NOT initializing for c51 because missingsnaps=6 && day-1-20160101=1 && day-1-20151201=0
[root@dslustre21 ~]# 

And the disks look fine:

[root@dslustre21 ~]# zfs list -t snapshot
NAME                                                                USED  AVAIL  REFER  MOUNTPOINT
nexsanpool1/axial@day-1-20160101                                   7.39M      -  12.1G  -
nexsanpool1/axial@day-2-20160102                                   5.00M      -  12.1G  -
nexsanpool1/axial@day-3-20160103                                   1.53M      -  12.1G  -
nexsanpool1/axial@day-4-20160104                                       0      -  12.5G  -
nexsanpool1/c51@day-1-20160101                                     16.0K      -   922M  -
nexsanpool1/c51@day-2-20160102                                     16.0K      -   922M  -
nexsanpool1/c51@day-3-20160103                                     16.0K      -   922M  -
nexsanpool1/c51@day-4-20160104                                         0      -   922M  -
nexsanpool1/charmonium@day-1-20160101                              16.0K      -   272K  -
nexsanpool1/charmonium@day-2-20160102                              16.0K      -   272K  -
nexsanpool1/charmonium@day-3-20160103                              16.0K      -   272K  -
nexsanpool1/charmonium@day-4-20160104                                  0      -   272K  -
nexsanpool1/disco@day-1-20160101                                       0      -  6.29G  -
nexsanpool1/fourpluseight@day-1-20160101                               0      -  22.1G  -
nexsanpool1/heavylight@day-1-20160101                                  0      -  2.81T  -
nexsanpool1/hpqcd@day-1-20160101                                       0      -   778G  -
nexsanpool1/mixbk@day-1-20160101                                       0      -   268G  -
nexsanpool1/nonperturbativestaggered@day-1-20160101                    0      -  39.5G  -
nexsanpool1/ok-action@day-1-20160101                                   0      -  8.79G  -
nexsanpool1/rbcdwf@day-1-20160101                                      0      -   499G  -
nexsanpool1/rhqbbar@day-1-20160101                                     0      -   731G  -
nexsanpool2/axial@day-1-20160101                                    296K      -  12.1G  -
nexsanpool2/c51@day-1-20160101                                      176K      -   922M  -
nexsanpool2/charmonium@day-1-20160101                                  0      -   272K  -
nexsanpool2/disco@day-1-20160101                                       0      -  6.29G  -
nexsanpool2/fourpluseight@day-1-20160101                               0      -  22.1G  -
nexsanpool2/heavylight@day-1-20160101                                  0      -  2.81T  -
nexsanpool2/hpqcd@day-1-20160101                                       0      -   778G  -
nexsanpool2/mixbk@day-1-20160101                                       0      -   268G  -
nexsanpool2/nonperturbativestaggered@day-1-20160101                    0      -  39.5G  -
nexsanpool2/ok-action@day-1-20160101                                   0      -  8.79G  -
nexsanpool2/rbcdwf@day-1-20160101                                      0      -   499G  -
nexsanpool2/rhqbbar@day-1-20160101                                     0      -   731G  -
projectzfspool/projectzfs/axial@day-1-20160101                     5.66M      -  11.9G  -
projectzfspool/projectzfs/axial@day-2-20160102                     3.62M      -  11.9G  -
projectzfspool/projectzfs/axial@day-3-20160103                      792K      -  11.9G  -
projectzfspool/projectzfs/axial@day-4-20160104                      476K      -  12.3G  -
projectzfspool/projectzfs/axial@day-5-20160105                      492K      -  12.3G  -
projectzfspool/projectzfs/axial@day-6-20160106                     3.11M      -  12.3G  -
projectzfspool/projectzfs/c51@day-1-20160101                           0      -   747M  -
projectzfspool/projectzfs/c51@day-2-20160102                           0      -   747M  -
projectzfspool/projectzfs/c51@day-3-20160103                           0      -   747M  -
projectzfspool/projectzfs/c51@day-4-20160104                           0      -   747M  -
projectzfspool/projectzfs/c51@day-5-20160105                           0      -   747M  -
projectzfspool/projectzfs/c51@day-6-20160106                           0      -   747M  -
projectzfspool/projectzfs/charmonium@day-1-20160101                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-2-20160102                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-3-20160103                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-4-20160104                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-5-20160105                    0      -   136K  -
projectzfspool/projectzfs/charmonium@day-6-20160106                    0      -   136K  -
projectzfspool/projectzfs/disco@day-1-20160101                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-2-20160102                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-3-20160103                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-4-20160104                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-5-20160105                         0      -  10.7G  -
projectzfspool/projectzfs/disco@day-6-20160106                         0      -  10.7G  -
projectzfspool/projectzfs/fourpluseight@day-1-20160101                 0      -  26.5G  -
projectzfspool/projectzfs/fourpluseight@day-2-20160102                 0      -  26.5G  -
projectzfspool/projectzfs/fourpluseight@day-3-20160103                 0      -  26.5G  -
projectzfspool/projectzfs/fourpluseight@day-4-20160104                 0      -  26.5G  -
projectzfspool/projectzfs/fourpluseight@day-5-20160105                 0      -  26.5G  -
projectzfspool/projectzfs/fourpluseight@day-6-20160106                 0      -  26.5G  -
projectzfspool/projectzfs/heavylight@day-1-20160101                12.7G      -  3.07T  -
projectzfspool/projectzfs/heavylight@day-2-20160102                27.7M      -  3.07T  -
projectzfspool/projectzfs/heavylight@day-3-20160103                39.3M      -  3.07T  -
projectzfspool/projectzfs/heavylight@day-4-20160104                36.6M      -  3.07T  -
projectzfspool/projectzfs/heavylight@day-5-20160105                37.6M      -  3.07T  -
projectzfspool/projectzfs/heavylight@day-6-20160106                43.1M      -  3.08T  -
projectzfspool/projectzfs/hpqcd@day-1-20160101                         0      -   506G  -
projectzfspool/projectzfs/hpqcd@day-2-20160102                         0      -   506G  -
projectzfspool/projectzfs/hpqcd@day-3-20160103                         0      -   506G  -
projectzfspool/projectzfs/hpqcd@day-4-20160104                         0      -   506G  -
projectzfspool/projectzfs/hpqcd@day-5-20160105                         0      -   506G  -
projectzfspool/projectzfs/hpqcd@day-6-20160106                         0      -   506G  -
projectzfspool/projectzfs/mixbk@day-1-20160101                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-2-20160102                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-3-20160103                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-4-20160104                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-5-20160105                         0      -   487G  -
projectzfspool/projectzfs/mixbk@day-6-20160106                         0      -   487G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-1-20160101      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-2-20160102      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-3-20160103      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-4-20160104      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-5-20160105      0      -   122G  -
projectzfspool/projectzfs/nonperturbativestaggered@day-6-20160106      0      -   122G  -
projectzfspool/projectzfs/ok-action@day-1-20160101                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-2-20160102                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-3-20160103                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-4-20160104                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-5-20160105                     0      -  12.6G  -
projectzfspool/projectzfs/ok-action@day-6-20160106                     0      -  12.6G  -
projectzfspool/projectzfs/rbcdwf@day-1-20160101                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-2-20160102                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-3-20160103                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-4-20160104                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-5-20160105                        0      -   572G  -
projectzfspool/projectzfs/rbcdwf@day-6-20160106                        0      -   572G  -
projectzfspool/projectzfs/rhqbbar@day-1-20160101                   31.6M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-2-20160102                   7.52M      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-3-20160103                    216K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-4-20160104                    104K      -   818G  -
projectzfspool/projectzfs/rhqbbar@day-5-20160105                    512K      -   801G  -
projectzfspool/projectzfs/rhqbbar@day-6-20160106                    392K      -   801G  -
[root@dslustre21 ~]# zpool list
NAME             SIZE  ALLOC   FREE    CAP  DEDUP  HEALTH  ALTROOT
nexsanpool1     16.2T  7.69T  8.56T    47%  1.00x  ONLINE  -
nexsanpool2     16.2T  13.0T  3.21T    80%  1.00x  ONLINE  -
projectzfspool  7.25T  5.61T  1.64T    77%  1.00x  ONLINE  -
[root@dslustre21 ~]# 

Since I didn't add the full path, the daily cron that does the replicas is not working, will fix the system calls in the scripts now...

#30 Updated by Gerard Bernabeu Altayo over 4 years ago

Still need to:

1. The script that generates the gzips should also generate a file (%Y%m.adler32) with the CRC of them, for future comparison in tape. Probably I can use http://fibrevillage.com/scripting/107-how-to-calculate-adler32-in-perl

2. write a script that should run manually to:

1. Check for existance of all files from the list (1) on local disk and pnfs, if exists:
2. Compare its crc with the one stored by enstore*
3. if the crc matches, Remove the files from dslustre21 

*We can get the CRC (adler32, seed=1 according to Chih-Hao) from enstore with:

[root@lqcdsrm 2015-12-30]# source /home/enstore/setup-enstore 
[root@lqcdsrm 2015-12-30]# enstore pnfs --xref /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-day-1-20151201.gz
volume: VPK371
location_cookie: 0000_000000000_0000173
size: 8640222920
file_family: project_backup_B
original_name: /pnfs/lqcd/backups/lqcdproj_backups/project_backup_B/2015-12-30/axial-day-1-20151201.gz
map_file: 
pnfsid_file: 000003BF82C308C647909617EA77F981F302
pnfsid_map: 
bfid: CDMS145149698500000
origdrive: enmvr095:/dev/rmt/tps5d0n:579004001915
crc: 3398356087
[root@lqcdsrm 2015-12-30]# 

#31 Updated by Gerard Bernabeu Altayo over 4 years ago

  • Status changed from New to Resolved

this task is now fully automated, closing.



Also available in: Atom PDF