Bug #24402
Not all the logfiles are rotated in the factory
Start date:
05/12/2020
Due date:
% Done:
0%
Estimated time:
First Occurred:
Occurs In:
Stakeholders:
Factory Ops
Description
This is the log configuation in the factory:
<log_retention> <condor_logs max_days="14.0" max_mbytes="100.0" min_days="3.0"/> <job_logs max_days="7.0" max_mbytes="100.0" min_days="3.0"/> <summary_logs max_days="31.0" max_mbytes="100.0" min_days="3.0"/> <process_logs> <process_log extension="info" max_days="7.0" max_mbytes="100.0" min_days="3.0" msg_types="INFO"/> <process_log extension="err" max_days="7.0" max_mbytes="100.0" min_days="3.0" msg_types="DEBUG,ERR,WARN,EXCEPTION"/> </process_logs> </log_retention>
However, it looks like some of the entry logfiles never rotates:
/var/log/gwms-factory/server/entry_Gluex_US_NUMEP_grid1/: total 721832 -rw-r--r-- 1 gfactory gfactory 104861331 6 feb 17.57 Gluex_US_NUMEP_grid1.info.log.2020-02-06_17-57 -rw-r--r-- 1 gfactory gfactory 104860362 21 feb 16.35 Gluex_US_NUMEP_grid1.info.log.2020-02-21_16-35 -rw-r--r-- 1 gfactory gfactory 104858829 10 mar 23.24 Gluex_US_NUMEP_grid1.info.log.2020-03-10_23-24 -rw-r--r-- 1 gfactory gfactory 104857800 14 mar 00.19 Gluex_US_NUMEP_grid1.err.log.2020-03-14_00-19 -rw-r--r-- 1 gfactory gfactory 104858131 29 mar 04.48 Gluex_US_NUMEP_grid1.info.log.2020-03-29_04-48 -rw-r--r-- 1 gfactory gfactory 104861166 22 apr 07.00 Gluex_US_NUMEP_grid1.info.log.2020-04-22_07-00
The old OSG production factory used to have a tmpwatch in a cronjob to clean them, but the new one run into disk issues because of these files never rotating. Also, check if files are rotated for disabled entries as well.
History
#1 Updated by Marco Mambelli 6 months ago
- Target version changed from v3_6_4 to v3_6_5
#2 Updated by Marco Mambelli 5 months ago
- Target version changed from v3_6_5 to v3_6_6
#3 Updated by Marco Mambelli 3 months ago
- Target version changed from v3_6_6 to v3_6_7
#4 Updated by Marco Mambelli 20 days ago
- Target version changed from v3_6_7 to v3_7_4