Project

General

Profile

Bug #14886

Memory leaks

Added by Marc Mengel almost 4 years ago. Updated almost 4 years ago.

Status:
Closed
Priority:
Urgent
Assignee:
Target version:
Start date:
12/23/2016
Due date:
% Done:

100%

Estimated time:
First Occurred:
Scope:
Internal
Experiment:
-
Stakeholders:
Duration:

Description

THe production stuff is growing memory massively again; so reviewing for possible memory leaks.

Read a big writeup on https://github.com/kennethreitz/requests/issues/1685 about leaks using the requests library etc. doing http calls out. Concluded that there are two places we have possible issues: 1)failing to close() http request results, and 2) the exception leaks we've seen before. So I went through various routines to make sure everywere we do an http get or post we close the result object, even in exception handlers.

Apparently just letting http connection objects get destroyed does not call all the close() code. :-(.

History

#1 Updated by Marc Mengel almost 4 years ago

Changes in c9bb367

#2 Updated by Marc Mengel almost 4 years ago

  • Status changed from Work in progress to Resolved
  • % Done changed from 0 to 100

Development instance of jobsub_q_scraper is steady at 219m vsize overnight since this cleanup (yay!).

The uwsgi threads in development do not seem to be obviously leaking memory anymore either, although

#3 Updated by Marc Mengel almost 4 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF