Project

General

Profile

Bug #12945

Excessive memory use in ROOT 6 build of uboonecode

Added by Marc Paterno about 3 years ago. Updated about 3 years ago.

Status:
Closed
Priority:
High
Assignee:
Category:
Application
Target version:
Start date:
06/16/2016
Due date:
% Done:

100%

Estimated time:
Spent time:
Occurs In:
Scope:
Internal
Experiment:
MicroBooNE
SSI Package:
art
Duration:

Description

The following is the original report, from email by Herb Greenlee.

Since I've heard from Lynn that larsoft wants to migrate to larsoft v06 this week, here are some tests of memory footprint of v06_00_00_rc4.

I tested the full mc+reco chain. I got memory problems getting jobs to run in the reco1 stage (fcl reco_uboone_mcc7_fullrawdigits_driver_stage1.fcl) on bnb+cosmic mc events. Reco1 (and reco2) jobs wouldn't run on grid without requesting larger than default memory.

Peak memory of reco1 jobs in v05_12_01 about 1.7 GB.
Peak memory of reco1 jobs in v06_00_00_rc4 about 3.8 GB.

More than a factor of two increase in memory. This conflicts with statements that the root 6 memory increase is "a few hundre MB" which I heard from art team members in meetings. Unless indeed there are remaining bugs which are causing different results, which of course is a possibility. However, I don't see significant differences in size of data products between root 6 and root 5.

Looking at Per-event summary of MemoryTracker, I noticed a roughly linear increase memory use of 200 MB/event, which smells like a leak (but there don't seem to be leaks in root 5?).

Larsoft v06_00_00_rc4 is compiled against art v2_00_02. According to art release nodes, memory optimizations are included in this version (actually already in art v2_00_01).

Bottom line, I don't think larsoft v06 is ready, and microboone doesn't sign off.

root-5-6-rss-usage.pdf (4.87 KB) root-5-6-rss-usage.pdf Marc Paterno, 06/16/2016 05:04 PM
root-5-6-difference-by-event.pdf (4.69 KB) root-5-6-difference-by-event.pdf Marc Paterno, 06/16/2016 05:04 PM

History

#1 Updated by Marc Paterno about 3 years ago

  • Status changed from New to Accepted

We are analyzing the problem. We are using the input files provided by Herb, and comparing ROOT5-based and ROOT6-based versions of art.

#2 Updated by Marc Paterno about 3 years ago

Herb, if you have the MemoryTracker database output from the job from which you quoted the memory increase, please send it to us. If you also have the TimeTracker database output, please send that as well.

#3 Updated by Marc Paterno about 3 years ago

We have run several jobs, one using ROOT5 and one using ROOT6, reading one the files provided by Herb. In our jobs we ran only ROOT input and ROOT output (not using any MicroBooNE modules), to look at how ROOT i/o may have changed. We have turned off fast cloning, because that is how the real jobs were run.

Attached are two plots. One shows no event-by-event memory usage increase either in ROOT 5 or ROOT 6. The other shows that the difference in memory (RSS) usage between ROOT5 and ROOT 6 is on the order of 200MB.

Despite not being relevant for grid memory limits, we also looked at Vsize measurements, and they also show no increase event-by-event.

We are currently attempting to run the full job, including all the MicroBooNE modules and running over all the data files, to look at the memory tracker output.

#4 Updated by Lynn Garren about 3 years ago

Thanks to Paul Russo, the bug is fixed in root v6_06_04b. A secondary workaround was also made in art itself.
art v2_00_03, nutools v2_00_01, and larsoftobj v1_02_02 are now available with the new releases of art, canvas, and gallery. larsoft v06_00_00_rc5 is tagged and building.

#5 Updated by Kyle Knoepfel about 3 years ago

  • Status changed from Accepted to Resolved
  • Assignee set to Paul Russo
  • % Done changed from 0 to 100

#6 Updated by Kyle Knoepfel about 3 years ago

  • Status changed from Resolved to Closed

#7 Updated by Kyle Knoepfel about 3 years ago

  • Target version set to 2.00.03


Also available in: Atom PDF