Project

General

Profile

MemoryTracker (versions older than art 2.06)

Optional service, not supported on Mac OSX systems.

Updates in progress.

Configuration

The allowed configuration is:

services: {
  MemoryTracker: {

    ignoreTotal       : [[ integer ]] # no. of events to exclude in summary - '1' by default

    printSummaries    : ["*"] # or any combination of
                        ["general","event","module"]

    includeMallocInfo : [[ true | false ]]

    dbOutput : {

      filename : "your_sqlite_db_name.db" # empty name by default -- implies in-memory databse

      overwrite: [[ true | false ]] # overwrite existing database file - 'false' by default
    }

  }
}

The default value of 1 for the ignoreTotal parameter is chosen since the first event often yields a large increase in resident or virtual memory, which doesn't necessarily correspond to a memory problem. Although the first event will be recorded in the EventInfo table in the database file, it is excluded from the summary printout. The wildcard in printSummaries will print all general, per-event, and per-module information to any specified messagefacility destinations. If the user specifies an empty list, no summaries are logged to the messagefacility destinations, although the general summary will be logged in the sqlite database.

Malloc info can be accessed only in the sqlite database. Per-event and -module information is included that reflects the mallinfo(3) Linux man page. If an empty filename is specified in conjunction with a true value of includeMallocInfo, that is a configuration error and an exception is thrown.

messagefacility logged message

A full summary, printed to any configured messagefacility destinations, looks like:

====================================================================================================

MemoryTracker General SUMMARY (all numbers in units of Mbytes)

  Peak virtual memory usage (VmPeak)  : 498.355 Mbytes
  Peak resident set size usage (VmHWM): 141.547 Mbytes

ProcessStep          Module ID                                Δ Vsize      Δ RSS
===================================================================================
Module Construction  a1:TestSimpleMemoryCheckProducer            0         0.012
Module Construction  a2:TestSimpleMemoryCheckProducer            0           0
Module Construction  a3:TestSimpleMemoryCheckProducer            0           0
Module Construction  TriggerResults:TriggerResultInserter      4.059       0.207
-----------------------------------------------------------------------------------
Module beginJob      a1:TestSimpleMemoryCheckProducer            0           0
Module beginJob      a2:TestSimpleMemoryCheckProducer            0           0
Module beginJob      a3:TestSimpleMemoryCheckProducer            0           0
Module beginJob      TriggerResults:TriggerResultInserter        0           0
-----------------------------------------------------------------------------------
Module beginRun      a1:TestSimpleMemoryCheckProducer            0           0
Module beginRun      a2:TestSimpleMemoryCheckProducer            0           0
Module beginRun      a3:TestSimpleMemoryCheckProducer            0           0
Module beginRun      TriggerResults:TriggerResultInserter        0           0
-----------------------------------------------------------------------------------
Module beginSubRun   a1:TestSimpleMemoryCheckProducer            0         0.008
Module beginSubRun   a2:TestSimpleMemoryCheckProducer            0           0
Module beginSubRun   a3:TestSimpleMemoryCheckProducer            0           0
Module beginSubRun   TriggerResults:TriggerResultInserter        0           0
-----------------------------------------------------------------------------------
Module endSubRun     a1:TestSimpleMemoryCheckProducer            0           0
Module endSubRun     a2:TestSimpleMemoryCheckProducer            0           0
Module endSubRun     a3:TestSimpleMemoryCheckProducer            0           0
Module endSubRun     TriggerResults:TriggerResultInserter        0           0
-----------------------------------------------------------------------------------
Module endRun        a1:TestSimpleMemoryCheckProducer            0           0
Module endRun        a2:TestSimpleMemoryCheckProducer            0           0
Module endRun        a3:TestSimpleMemoryCheckProducer            0           0
Module endRun        TriggerResults:TriggerResultInserter        0           0
-----------------------------------------------------------------------------------
Module endJob        a1:TestSimpleMemoryCheckProducer            0           0
Module endJob        a2:TestSimpleMemoryCheckProducer            0           0
Module endJob        a3:TestSimpleMemoryCheckProducer            0           0
Module endJob        TriggerResults:TriggerResultInserter        0           0

====================================================================================================

MemoryTracker Per-event SUMMARY

Events increasing Vsize (Mbytes)        Vsize      Δ Vsize      RSS         Δ RSS
====================================================================================
  run: 1 subRun: 0 event: 5            492.609      1.148      135.688      1.148
  run: 1 subRun: 0 event: 6            493.758      1.148      136.840      1.148
  run: 1 subRun: 0 event: 7            494.906      1.148      137.992      1.148
  run: 1 subRun: 0 event: 8            496.055      1.148      139.141      1.148

Events with large Vsize (Mbytes)        Vsize      Δ Vsize      RSS         Δ RSS
====================================================================================
  run: 1 subRun: 0 event: 9            497.203      1.148      140.293      1.148
  run: 1 subRun: 0 event: 8            496.055      1.148      139.141      1.148
  run: 1 subRun: 0 event: 7            494.906      1.148      137.992      1.148
  run: 1 subRun: 0 event: 6            493.758      1.148      136.840      1.148

====================================================================================================

MemoryTracker Per-module SUMMARY

Modules increasing Vsize (Mbytes)            Vsize      Δ Vsize      RSS         Δ RSS
===========================================================================================
p1:a1:TestSimpleMemoryCheckProducer
  run: 1 subRun: 0 event: 5                 491.844      0.383      134.922      0.383
  run: 1 subRun: 0 event: 6                 492.992      0.383      136.074      0.383
  run: 1 subRun: 0 event: 7                 494.141      0.383      137.227      0.383
  run: 1 subRun: 0 event: 8                 495.289      0.383      138.375      0.383
-------------------------------------------------------------------------------------------
p1:a2:TestSimpleMemoryCheckProducer
  run: 1 subRun: 0 event: 4                 491.078      0.383      134.152      0.383
  run: 1 subRun: 0 event: 5                 492.227      0.383      135.305      0.383
  run: 1 subRun: 0 event: 6                 493.375      0.383      136.457      0.383
  run: 1 subRun: 0 event: 7                 494.523      0.383      137.609      0.383
-------------------------------------------------------------------------------------------
p1:a3:TestSimpleMemoryCheckProducer
  run: 1 subRun: 0 event: 4                 491.461      0.383      134.535      0.383
  run: 1 subRun: 0 event: 5                 492.609      0.383      135.688      0.383
  run: 1 subRun: 0 event: 6                 493.758      0.383      136.840      0.383
  run: 1 subRun: 0 event: 7                 494.906      0.383      137.992      0.383
-------------------------------------------------------------------------------------------
p1:TriggerResults:TriggerResultInserter
  [[ no events ]]

Modules with large Vsize (Mbytes)            Vsize      Δ Vsize      RSS         Δ RSS
===========================================================================================
p1:a1:TestSimpleMemoryCheckProducer
  run: 1 subRun: 0 event: 9                 496.438      0.383      139.527      0.383
  run: 1 subRun: 0 event: 8                 495.289      0.383      138.375      0.383
  run: 1 subRun: 0 event: 7                 494.141      0.383      137.227      0.383
  run: 1 subRun: 0 event: 6                 492.992      0.383      136.074      0.383
-------------------------------------------------------------------------------------------
p1:a2:TestSimpleMemoryCheckProducer
  run: 1 subRun: 0 event: 9                 496.820      0.383      139.910      0.383
  run: 1 subRun: 0 event: 8                 495.672      0.383      138.758      0.383
  run: 1 subRun: 0 event: 7                 494.523      0.383      137.609      0.383
  run: 1 subRun: 0 event: 6                 493.375      0.383      136.457      0.383
-------------------------------------------------------------------------------------------
p1:a3:TestSimpleMemoryCheckProducer
  run: 1 subRun: 0 event: 9                 497.203      0.383      140.293      0.383
  run: 1 subRun: 0 event: 8                 496.055      0.383      139.141      0.383
  run: 1 subRun: 0 event: 7                 494.906      0.383      137.992      0.383
  run: 1 subRun: 0 event: 6                 493.758      0.383      136.840      0.383
-------------------------------------------------------------------------------------------
p1:TriggerResults:TriggerResultInserter
  run: 1 subRun: 0 event: 9                 497.203        0        140.293        0
  run: 1 subRun: 0 event: 8                 496.055        0        139.141        0
  run: 1 subRun: 0 event: 7                 494.906        0        137.992        0
  run: 1 subRun: 0 event: 6                 493.758        0        136.840        0

====================================================================================================

The Vsize and RSS values correspond to memory usage captured at the end of each event.

sqlite database

[ For SQLite help, see here. ]

The sqlite database has up to five tables:
  • PeakUsage
  • OtherInfo
  • EventInfo
  • ModuleInfo
  • EventMallocInfo [ optional ]
  • ModuleMallocInfo [ optional ]

An example table of EventInfo would look like:

Run         SubRun      Event       Vsize        DeltaVsize  RSS          DeltaRSS  
----------  ----------  ----------  -----------  ----------  -----------  ----------
1           0           1           494.3515625  0           135.2890625  1.203125  
1           0           2           494.3515625  0           136.4492187  1.14453125
1           0           3           495.5        1.1484375   137.6015625  1.1484375 
[ etc. ]

An example table of ModuleInfo would be:

Run         SubRun      Event       Path        ModuleLabel  ModuleType                     Vsize        DeltaVsize  RSS          DeltaRSS  
----------  ----------  ----------  ----------  -----------  -----------------------------  -----------  ----------  -----------  ----------
1           0           1           p1          a1           TestSimpleMemoryCheckProducer  494.3515625  0           134.4765625  0.3828125 
1           0           1           p1          a2           TestSimpleMemoryCheckProducer  494.3515625  0           134.8632812  0.3828125 
1           0           1           p1          a3           TestSimpleMemoryCheckProducer  494.3515625  0           135.2460937  0.3828125 
1           0           1           p1          TriggerResu  TriggerResultInserter          494.3515625  0           135.2890625  0.04296875
1           0           2           p1          a1           TestSimpleMemoryCheckProducer  494.3515625  0           135.6835937  0.37890625
[ etc. ]      

If includeMallocInfo is true, then the *MallocInfo tables are filled according to the following schema:

ModRowId    arena       ordblks     keepcost    hblkhd      hblks       uordblks    fordblks  
----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------
1           10641408    11          729184      0           0           9910864     730544    
2           10641408    7           329168      0           0           10310992    330416    
3           10641408    7           329168      401408      1           10311088    330320    
4           10641408    4           329168      401408      1           10311600    329808    
5           10641408    12          326752      802816      2           10311392    330016    
6           10641408    7           328000      1204224     3           10311536    329872    
7           10641408    7           328000      1605632     4           10311664    329744    
[ etc. ]

where ModRowId specifies the row number of the corresponding entry in the ModuleInfo table. An EvtRowId value is specified for the EventMallocInfo table, mapping the mallinfo data to the entries in the EventInfo table.

The PeakUsage table contains simple information about the job:

Name        Value         Description             
----------  ------------  ---------------------------
VmPeak      498.35546875  Peak virtual memory (MB)
VmHWM       141.546875    Peak resident set size (MB)

The OtherInfo contains memory for module construction and the {begin,end}Run and {begin,end}SubRun callbacks:

ProcessStep           ModuleLabel      ModuleType                     DeltaVsize  DeltaRSS  
--------------------  ---------------  -----------------------------  ----------  ----------
Module Construction   a1               TestSimpleMemoryCheckProducer  0           0.01171875
Module Construction   a2               TestSimpleMemoryCheckProducer  0           0         
Module Construction   a3               TestSimpleMemoryCheckProducer  0           0         
Module Construction   TriggerResults   TriggerResultInserter          4.05859375  0.20703125
Module beginJob       a1               TestSimpleMemoryCheckProducer  0           0         
Module beginJob       a2               TestSimpleMemoryCheckProducer  0           0         
Module beginJob       a3               TestSimpleMemoryCheckProducer  0           0         
Module beginJob       TriggerResults   TriggerResultInserter          0           0         
Module beginRun       a1               TestSimpleMemoryCheckProducer  0           0         
Module beginRun       a2               TestSimpleMemoryCheckProducer  0           0         
Module beginRun       a3               TestSimpleMemoryCheckProducer  0           0         
Module beginRun       TriggerResults   TriggerResultInserter          0           0         
Module beginSubRun    a1               TestSimpleMemoryCheckProducer  0           0.0078125 
Module beginSubRun    a2               TestSimpleMemoryCheckProducer  0           0         
Module beginSubRun    a3               TestSimpleMemoryCheckProducer  0           0         
Module beginSubRun    TriggerResults   TriggerResultInserter          0           0         
Module endSubRun      a1               TestSimpleMemoryCheckProducer  0           0         
Module endSubRun      a2               TestSimpleMemoryCheckProducer  0           0         
Module endSubRun      a3               TestSimpleMemoryCheckProducer  0           0         
Module endSubRun      TriggerResults   TriggerResultInserter          0           0         
Module endRun         a1               TestSimpleMemoryCheckProducer  0           0         
Module endRun         a2               TestSimpleMemoryCheckProducer  0           0         
Module endRun         a3               TestSimpleMemoryCheckProducer  0           0         
Module endRun         TriggerResults   TriggerResultInserter          0           0         
Module endJob         a1               TestSimpleMemoryCheckProducer  0           0         
Module endJob         a2               TestSimpleMemoryCheckProducer  0           0         
Module endJob         a3               TestSimpleMemoryCheckProducer  0           0         
Module endJob         TriggerResults   TriggerResultInserter          0           0