Project

General

Profile

Bug #24402

Not all the logfiles are rotated in the factory

Added by Marco Mascheroni about 2 months ago.

Status:
New
Priority:
Normal
Assignee:
Marco Mascheroni
Category:
-
Target version:
v3_6_4
Start date:
05/12/2020
Due date:
% Done:

0%

Estimated time:
First Occurred:
Occurs In:
Stakeholders:

Factory Ops

Duration:

Description

This is the log configuation in the factory:

   <log_retention>
      <condor_logs max_days="14.0" max_mbytes="100.0" min_days="3.0"/>
      <job_logs max_days="7.0" max_mbytes="100.0" min_days="3.0"/>
      <summary_logs max_days="31.0" max_mbytes="100.0" min_days="3.0"/>
      <process_logs>
                <process_log extension="info" max_days="7.0" max_mbytes="100.0" min_days="3.0" msg_types="INFO"/>
                <process_log extension="err" max_days="7.0" max_mbytes="100.0" min_days="3.0" msg_types="DEBUG,ERR,WARN,EXCEPTION"/>
      </process_logs>
   </log_retention>

However, it looks like some of the entry logfiles never rotates:

/var/log/gwms-factory/server/entry_Gluex_US_NUMEP_grid1/:
total 721832
-rw-r--r--   1 gfactory gfactory 104861331  6 feb 17.57 Gluex_US_NUMEP_grid1.info.log.2020-02-06_17-57
-rw-r--r--   1 gfactory gfactory 104860362 21 feb 16.35 Gluex_US_NUMEP_grid1.info.log.2020-02-21_16-35
-rw-r--r--   1 gfactory gfactory 104858829 10 mar 23.24 Gluex_US_NUMEP_grid1.info.log.2020-03-10_23-24
-rw-r--r--   1 gfactory gfactory 104857800 14 mar 00.19 Gluex_US_NUMEP_grid1.err.log.2020-03-14_00-19
-rw-r--r--   1 gfactory gfactory 104858131 29 mar 04.48 Gluex_US_NUMEP_grid1.info.log.2020-03-29_04-48
-rw-r--r--   1 gfactory gfactory 104861166 22 apr 07.00 Gluex_US_NUMEP_grid1.info.log.2020-04-22_07-00

The old OSG production factory used to have a tmpwatch in a cronjob to clean them, but the new one run into disk issues because of these files never rotating. Also, check if files are rotated for disabled entries as well.

Also available in: Atom PDF