cet-is » art

The time estimate is for the initial analysis and diagnosis only.

Due to the elevated priority of the upcoming release, the target version for this issue has been adjusted.

We have investigated this issue at length, and we believe we understand what is happening. According to an allinea map profile, the memory grows linearly at a gradual rate until the end of the job, where a spike of roughly 140 MB is seen. The memory growth results from calls to TTree::Fill() and TBranch::Fill(). The default ROOT behavior, which art uses, is to impose no memory limit on any trees that are being filled. After each call to Fill(), a copy of the data is made to the TBasket buffer, the memory of which is not deallocated until the end of the process.

Gradual memory growth¶

[ Update: The reason for the gradual memory growth as stated here is now known to be incorrect. See #4946-9 below. ]

The gradual memory growth results from calls to TTree::Fill() at the per-event level. As there is no memory limit on the TTree objects in art, the memory grows until the end of the process. There are ways to flush/delete the cached TBasket buffers, but the current ROOT interface requires setting a maximum virtual memory size per TTree. The appropriate value for this limit depends on the expected size of the stored objects in the TTree and can thus be difficult to estimate ahead of time. Ideally we would allow only one TBasket in memory per input/output branch. However, until such a feature is available (or devisable by us), the user can specify a hard-coded limit in the FHiCL file:

outputs: {
  out1: {
    module_type : RootOutput
    fileName : "out1.root" 
    treeMaxVirtualSize : 50000000 # 50e6-byte limit
  }
}

When the virtual memory (as measured by ROOT) of the TBasket buffers reaches that limit, the buffers flushed, and the cached buffers and associated memory are deallocated.

140 MB spike at end¶

The spike at the end results from writing the art::FileIndex metadata to the output file. The FileIndex contains a vector of indices that is used for accessing the events in a downstream process. We estimate that the stored number of art::FileIndex bytes per event is on the order of 50 bytes. Running over the set of files you pointed us to, that amounts to roughly 66 MB. Before the data are compressed, ROOT copies the data, bringing it up to 132 MB or so. Accounting for the compression map then brings things in line with the observed 140 MB spike. The eventual compression on the art::FileIndex, however, is a factor of 8 or so, resulting in an on-disk art::FileIndex of roughly 8 MB (or 6 bytes/event).

Note that the number of subruns is largely immaterial. This spike at the end is primarily a linear function of the number of events that are stored in the final output.

The reason for the spike is that the entire art::FileIndex is written to disk at one time during one call to TBranch::Fill(). This is consistent with how the other metadata are stored. However, in order to suppress such a memory spike at the end, we are willing to create a new TTree object corresponding to the art::FileIndex, whose branch is of the type of entry stored in the art::FileIndex. We can then fill the branch one entry at a time, and flush/deallocate cached buffers whenever the size exceeds the expected basket size: on the order of 16 KB. The spike should then be immaterial compared to the other memory requirements of the job.

The time estimate of 40 hours corresponds only to the time estimated to address the memory spike at the end. Addressing the gradual memory growth issue will take more effort.

The memory spike at the end of the process has been removed by placing the art::FileIndex inside a TTree object with TBranch type of art::FileIndex::Element. A new fileFormatVersion was necessary for this change, bumping the version from 6 to 7. art will correctly load the FileIndex in old files -- i.e. this change is backwards compatible. Note, however, that files produced with file format version 7 are not readable by older versions of art.

The total virtual memory usage for this job decreased from 462 MB to 329 MB.

We are continuing to explore the reason behind the gradual memory growth, which is partially understood at the moment.

Implemented with art:3998a8ec3ce0a78ebb3548fb9a3de4ada4444a36.

As discussed at the stakeholders meeting yesterday, the removal of the memory spike is sufficient to consider this issue resolved. An additional issue will be opened that corresponds specifically to the gradual memory growth, part of which has been difficult to assess. The analysis of that behavior up till this point is discussed in issue #8426.