Project

General

Profile

MemoryTracker (art versions 2.07 and newer)

Optional service, not supported on macOS systems.

Configuration

The allowed configuration is:

services: {
  MemoryTracker: {

    dbOutput: {
      filename: ""  # default
      overwrite: false  # default
    }

    includeMallocInfo: false  # default
  }
}

Detailed memory information for the job is stored in an SQLite database file. The default value for the filename parameter is empty, which implies that only the peak VmPeak and VmHWM values will be reported to the terminal at the end of the job. Per-module and -event information is available in the SQLite file. If an empty filename is specified in conjunction with a true value of includeMallocInfo, that is a configuration error and an exception is thrown.

End-of-job summary

For users who enable the MemoryTracker service with a database filename of 'memoryUsage.db', an end-of-job summary is logged to the messagefacility that looks like:

====================================================================================================
MemoryTracker summary (base-10 MB units used)

  Peak virtual memory usage (VmPeak)  : 461.07 MB
  Peak resident set size usage (VmHWM): 171.004 MB
  Details saved in: 'memoryUsage.db'
====================================================================================================

Updates in progress.

The Vsize and RSS values correspond to memory usage captured at the end of each event.

sqlite database

[ For SQLite help, see here. ]

The sqlite database has up to five tables:
  • PeakUsage
  • OtherInfo
  • EventInfo
  • ModuleInfo
  • EventMallocInfo [ optional ]
  • ModuleMallocInfo [ optional ]

An example table of EventInfo would look like:

Run         SubRun      Event       Vsize        DeltaVsize  RSS          DeltaRSS  
----------  ----------  ----------  -----------  ----------  -----------  ----------
1           0           1           494.3515625  0           135.2890625  1.203125  
1           0           2           494.3515625  0           136.4492187  1.14453125
1           0           3           495.5        1.1484375   137.6015625  1.1484375 
[ etc. ]

An example table of ModuleInfo would be:

Run         SubRun      Event       Path        ModuleLabel  ModuleType                     Vsize        DeltaVsize  RSS          DeltaRSS  
----------  ----------  ----------  ----------  -----------  -----------------------------  -----------  ----------  -----------  ----------
1           0           1           p1          a1           TestSimpleMemoryCheckProducer  494.3515625  0           134.4765625  0.3828125 
1           0           1           p1          a2           TestSimpleMemoryCheckProducer  494.3515625  0           134.8632812  0.3828125 
1           0           1           p1          a3           TestSimpleMemoryCheckProducer  494.3515625  0           135.2460937  0.3828125 
1           0           1           p1          TriggerResu  TriggerResultInserter          494.3515625  0           135.2890625  0.04296875
1           0           2           p1          a1           TestSimpleMemoryCheckProducer  494.3515625  0           135.6835937  0.37890625
[ etc. ]      

If includeMallocInfo is true, then the *MallocInfo tables are filled according to the following schema:

ModRowId    arena       ordblks     keepcost    hblkhd      hblks       uordblks    fordblks  
----------  ----------  ----------  ----------  ----------  ----------  ----------  ----------
1           10641408    11          729184      0           0           9910864     730544    
2           10641408    7           329168      0           0           10310992    330416    
3           10641408    7           329168      401408      1           10311088    330320    
4           10641408    4           329168      401408      1           10311600    329808    
5           10641408    12          326752      802816      2           10311392    330016    
6           10641408    7           328000      1204224     3           10311536    329872    
7           10641408    7           328000      1605632     4           10311664    329744    
[ etc. ]

where ModRowId specifies the row number of the corresponding entry in the ModuleInfo table. An EvtRowId value is specified for the EventMallocInfo table, mapping the mallinfo data to the entries in the EventInfo table.

The PeakUsage table contains simple information about the job:

Name        Value         Description             
----------  ------------  ---------------------------
VmPeak      498.35546875  Peak virtual memory (MB)
VmHWM       141.546875    Peak resident set size (MB)

The OtherInfo contains memory for module construction and the {begin,end}Run and {begin,end}SubRun callbacks:

ProcessStep           ModuleLabel      ModuleType                     DeltaVsize  DeltaRSS  
--------------------  ---------------  -----------------------------  ----------  ----------
Module Construction   a1               TestSimpleMemoryCheckProducer  0           0.01171875
Module Construction   a2               TestSimpleMemoryCheckProducer  0           0         
Module Construction   a3               TestSimpleMemoryCheckProducer  0           0         
Module Construction   TriggerResults   TriggerResultInserter          4.05859375  0.20703125
Module beginJob       a1               TestSimpleMemoryCheckProducer  0           0         
Module beginJob       a2               TestSimpleMemoryCheckProducer  0           0         
Module beginJob       a3               TestSimpleMemoryCheckProducer  0           0         
Module beginJob       TriggerResults   TriggerResultInserter          0           0         
Module beginRun       a1               TestSimpleMemoryCheckProducer  0           0         
Module beginRun       a2               TestSimpleMemoryCheckProducer  0           0         
Module beginRun       a3               TestSimpleMemoryCheckProducer  0           0         
Module beginRun       TriggerResults   TriggerResultInserter          0           0         
Module beginSubRun    a1               TestSimpleMemoryCheckProducer  0           0.0078125 
Module beginSubRun    a2               TestSimpleMemoryCheckProducer  0           0         
Module beginSubRun    a3               TestSimpleMemoryCheckProducer  0           0         
Module beginSubRun    TriggerResults   TriggerResultInserter          0           0         
Module endSubRun      a1               TestSimpleMemoryCheckProducer  0           0         
Module endSubRun      a2               TestSimpleMemoryCheckProducer  0           0         
Module endSubRun      a3               TestSimpleMemoryCheckProducer  0           0         
Module endSubRun      TriggerResults   TriggerResultInserter          0           0         
Module endRun         a1               TestSimpleMemoryCheckProducer  0           0         
Module endRun         a2               TestSimpleMemoryCheckProducer  0           0         
Module endRun         a3               TestSimpleMemoryCheckProducer  0           0         
Module endRun         TriggerResults   TriggerResultInserter          0           0         
Module endJob         a1               TestSimpleMemoryCheckProducer  0           0         
Module endJob         a2               TestSimpleMemoryCheckProducer  0           0         
Module endJob         a3               TestSimpleMemoryCheckProducer  0           0         
Module endJob         TriggerResults   TriggerResultInserter          0           0