Project

General

Profile

Feature #15657

Milestone #15372: art multi-threading phase 1

Feature #15374: Modifying all framework-provided services to be thread safe.

Make MemoryTracker service thread safe

Added by Kyle Knoepfel over 2 years ago. Updated over 2 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
02/24/2017
Due date:
% Done:

100%

Estimated time:
8.00 h
Spent time:
Scope:
Internal
Experiment:
-
SSI Package:
art
Duration:

Description

Relies on cet::Ntuple, which is not yet thread-safe. Analysis should be done to determine what meaning memory-tracking has in a multi-threaded environment.


Related issues

Blocked by cetlib - Feature #15672: Make SQLite cet::Ntuple facility thread safeClosed2017-02-27

History

#1 Updated by Kyle Knoepfel over 2 years ago

  • Blocked by Feature #15672: Make SQLite cet::Ntuple facility thread safe added

#2 Updated by Kyle Knoepfel over 2 years ago

Removed MemoryTracker header so that users cannot explicitly interact with the MemoryTracker service. All that remains is to make the MemoryTracker callbacks thread-safe, which relies inherently on cet::Ntuple. Analysis must still happen to determine if MemoryTracker is a meaningful service in a multi-threaded environment.

#3 Updated by Kyle Knoepfel over 2 years ago

  • % Done changed from 0 to 40

#4 Updated by Kyle Knoepfel over 2 years ago

  • % Done changed from 40 to 90

Primary feature implemented with commit art:c0b2d9ea. Some further investigation is warranted as to whether all memory retrievals from procfs are thread-safe. Will mark as resolved once that is verified.

#5 Updated by Kyle Knoepfel over 2 years ago

  • Status changed from Assigned to Resolved
  • % Done changed from 90 to 100

Although the previous commit did make the MemoryTracker service thread-safe, it recorded information per-schedule, which, unlike the TimeTracker, is unlikely to be helpful. Instead, I changed the SQLite database schema so that only memory snapshot information is recorded in the database. Any changes to the VSize or RSS must be calculated from the database.

Whenever the MemoryTracker is enabled the summary printed at the end of the art job now looks like:

====================================================

MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 535.593 MB
  Peak resident set size usage (VmHWM): 186.814 MB
  Details saved in: 'memoryTracker.db'

====================================================

If per-event or per-module information is desired, the stored database should be queried. Base on discussions with Herb Greenlee and Rob Kutschke, the per-event/module information that was emitted to the terminal was not necessarily all that helpful.

Rob brought up the possibilities of reporting to the terminal whenever the Vsize or RSS increases at the time the increases happens. He has not been the first to suggest reinstating that. Rob also mentioned that it might be helpful to report the module in which the peak Vsize or RSS is obtained. That would be non-trivial to do since we since the peak Vsize/RSS does not necessarily correspond to whenever the memory snapshots are recorded. Rather, we retrieve the the peak Vsize/RSS values once at endJob through /proc/status.

If it is desired to reinstate the full event- and module-level summary, or if it is desired to reinstate the increase-in-memory-usage reporting while the increase is happening, a feature request should be opened.

Feature resolves by commit art:13d2a866.

#6 Updated by Kyle Knoepfel over 2 years ago

  • Target version set to 1209

#7 Updated by Kyle Knoepfel over 2 years ago

  • Target version changed from 1209 to 2.07.01

#8 Updated by Kyle Knoepfel over 2 years ago

  • Status changed from Resolved to Closed


Also available in: Atom PDF