Project

General

Profile

Bug #3820

Factory log cleanup is broken

Added by Burt Holzman over 6 years ago. Updated over 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Parag Mhashilkar
Category:
Factory
Target version:
Start date:
05/10/2013
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
First Occurred:
Occurs In:
Stakeholders:
Duration:

Description


Hello glideinWMS Support,

We have evidence to suggest glideinWMS 2_7 is not removing all our logs as it should.  We have the following log retention policies set:
      <condor_logs max_days="14.0" max_mbytes="1000.0" min_days="3.0"/>
      <job_logs max_days="10.0" max_mbytes="1000.0" min_days="5.0"/>
      <logs max_days="7.0" max_mbytes="1000.0" min_days="3.0"/>
      <summary_logs max_days="31.0" max_mbytes="100.0" min_days="3.0"/>

However I looked in various places and was surprised to see various logs since April 10th still lying around, and that was the day we upgraded to 2_7.

Examples:
~/glideinsubmit/glidein_v2_0/log/factory:
-rw-rw-r-- 1 gfactory gfactory 3215938 Apr 10 23:59 group_0.20130410.debug.log
-rw-rw-r-- 1 gfactory gfactory   41931 Apr 10 22:31 group_0.20130410.err.log
-rw-rw-r-- 1 gfactory gfactory      60 Apr 10 14:22 factory.20130410.admin.log

~/glideinsubmit/glidein_v2_0/log/entry_CMS_T2_US_UCSD_gw2:
-rw-rw-r-- 1 gfactory gfactory  6025606 Apr 10 23:59 factory.20130410.info.log
-rw-rw-r-- 1 gfactory gfactory    97872 Apr 10 23:57 factory.20130410.err.log
-rw-rw-r-- 1 gfactory gfactory    53165 Apr 10 23:55 completed_jobs_20130410.log

~/glideinsubmit/glidein_v2_0/client_log/user_fecmsucsd/entry_CMS_T2_US_UCSD_gw2:
-rw-rw-r-- 1 fecmsucsd gwms  20061 Apr 10 12:42 job.491247.1.out
-rw-rw-r-- 1 fecmsucsd gwms  53659 Apr 10 12:39 job.491225.1.err
-rw-rw-r-- 1 fecmsucsd gwms  19787 Apr 10 12:39 job.491225.1.out

We are most concerned with the condor_logs (.err and .out of glideins) as that is the bulk of what we get written back on disk. As you can see the earliest of those logs based on our retention rules should be 14 days old at most, thus we shouldn't have anything before April 23rd.

I suspect the easiest way to investigate this will be for a gwms developer to log into our node and poke around.

Any help would be greatly appreciated.

Thanks,
Jeff Dost
OSG Glidein Factory Operations

Subtasks

Bug #3846: Master: Factory log cleanup is brokenClosedParag Mhashilkar

History

#1 Updated by Parag Mhashilkar over 6 years ago

  • Target version changed from v2_7_x to v2_7_1

#2 Updated by Parag Mhashilkar over 6 years ago

  • Status changed from Assigned to Feedback
  • Assignee changed from Parag Mhashilkar to Burt Holzman

Can you please review branch_v2plus_3820?

#3 Updated by Parag Mhashilkar over 6 years ago

This was reviewed and internally tested.

Python logging module used in master should handle things correctly by default, so not merging/cherry-picking into master.

Resolving the ticket.

#4 Updated by Parag Mhashilkar over 6 years ago

  • Status changed from Feedback to Closed
  • Assignee changed from Burt Holzman to Parag Mhashilkar


Also available in: Atom PDF