Project

General

Profile

Task #10378

Clean up /var/lib/puppet/clientbucket/ directory on store nodes.

Added by Chih-Hao Huang about 4 years ago. Updated about 4 years ago.

Status:
Assigned
Priority:
Normal
Start date:
10/02/2015
Due date:
10/09/2015
% Done:

0%

Estimated time:
8.00 h
Spent time:
Duration: 8

Description

Over the time, the cache accumulates in /var/lib/puppet/clientbucket/ directory. They are mainly /etc/grid-security/grid-mapfile.
On the nodes with only 40GB / partition, this has slowly become a problem while / partition is 80% full and the warning is showing up in check_mk.
Though 20% is still not disaster yet in the near future, it will be eventually.
For now, cleaning puppet cache directory would fix the problem and buy us some time (2 years?).
The longer term solution will be cleaning it regularly or disable the cache.
In addition, we need to get to the bottom of reducing the change to grid-mapfile if possible.
This issue was found on cmsstor401 and cmsstor402 but it affects all nodes (not limited to store nodes) that has grid-mapfile.

History

#1 Updated by Chih-Hao Huang about 4 years ago

  • Status changed from New to Assigned

cmsstor401 and cmsstor402 were reported in check_mk to have warning on / partition 80% full.
The root cause was /var/lib/puppet/clientbucket/ used 27GB (of 40GB partition).
The cached files were mainly /etc/grid-security/grid-mapfile.
Here is a complete log of the work on cmsstor402:

[root@cmsstor402 ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 41284928 30943804 8243972 79% /
tmpfs 32975340 0 32975340 0% /dev/shm
/dev/sda1 1032088 128804 850856 14% /boot
/dev/sda5 898341304 73668 852627804 1% /storage/local/data1
/dev/sdb 75409749120 26964339040 48445410080 36% /storage/data1
/dev/sdc 75409749120 26408989348 49000759772 36% /storage/data2
[root@cmsstor402 ~]# du -sh /var/lib/puppet/clientbucket/*
1.7G /var/lib/puppet/clientbucket/0
1.8G /var/lib/puppet/clientbucket/1
1.8G /var/lib/puppet/clientbucket/2
1.8G /var/lib/puppet/clientbucket/3
1.7G /var/lib/puppet/clientbucket/4
1.6G /var/lib/puppet/clientbucket/5
1.7G /var/lib/puppet/clientbucket/6
1.6G /var/lib/puppet/clientbucket/7
1.7G /var/lib/puppet/clientbucket/8
1.5G /var/lib/puppet/clientbucket/9
1.7G /var/lib/puppet/clientbucket/a
1.6G /var/lib/puppet/clientbucket/b
1.9G /var/lib/puppet/clientbucket/c
1.8G /var/lib/puppet/clientbucket/d
1.8G /var/lib/puppet/clientbucket/e
1.7G /var/lib/puppet/clientbucket/f
[root@cmsstor402 ~]# cd /var/lib/puppet/clientbucket
[root@cmsstor402 clientbucket]# find . -type f -mtime +7 -exec rm {} \;
[root@cmsstor402 clientbucket]# du -sh /var/lib/puppet/clientbucket/*
24M /var/lib/puppet/clientbucket/0
69M /var/lib/puppet/clientbucket/1
43M /var/lib/puppet/clientbucket/2
50M /var/lib/puppet/clientbucket/3
36M /var/lib/puppet/clientbucket/4
63M /var/lib/puppet/clientbucket/5
44M /var/lib/puppet/clientbucket/6
44M /var/lib/puppet/clientbucket/7
47M /var/lib/puppet/clientbucket/8
31M /var/lib/puppet/clientbucket/9
69M /var/lib/puppet/clientbucket/a
67M /var/lib/puppet/clientbucket/b
65M /var/lib/puppet/clientbucket/c
42M /var/lib/puppet/clientbucket/d
58M /var/lib/puppet/clientbucket/e
32M /var/lib/puppet/clientbucket/f
[root@cmsstor402 clientbucket]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda2 41284928 3721712 35466064 10% /
tmpfs 32975340 0 32975340 0% /dev/shm
/dev/sda1 1032088 128804 850856 14% /boot
/dev/sda5 898341304 73668 852627804 1% /storage/local/data1
/dev/sdb 75409749120 26972300776 48437448344 36% /storage/data1
/dev/sdc 75409749120 26408989348 49000759772 36% /storage/data2
[root@cmsstor402 clientbucket]#



Also available in: Atom PDF