Excessive memory use in ROOT 6 build of uboonecode
The following is the original report, from email by Herb Greenlee.
Since I've heard from Lynn that larsoft wants to migrate to larsoft v06 this week, here are some tests of memory footprint of v06_00_00_rc4.
I tested the full mc+reco chain. I got memory problems getting jobs to run in the reco1 stage (fcl reco_uboone_mcc7_fullrawdigits_driver_stage1.fcl) on bnb+cosmic mc events. Reco1 (and reco2) jobs wouldn't run on grid without requesting larger than default memory.
Peak memory of reco1 jobs in v05_12_01 about 1.7 GB.
Peak memory of reco1 jobs in v06_00_00_rc4 about 3.8 GB.
More than a factor of two increase in memory. This conflicts with statements that the root 6 memory increase is "a few hundre MB" which I heard from art team members in meetings. Unless indeed there are remaining bugs which are causing different results, which of course is a possibility. However, I don't see significant differences in size of data products between root 6 and root 5.
Looking at Per-event summary of MemoryTracker, I noticed a roughly linear increase memory use of 200 MB/event, which smells like a leak (but there don't seem to be leaks in root 5?).
Larsoft v06_00_00_rc4 is compiled against art v2_00_02. According to art release nodes, memory optimizations are included in this version (actually already in art v2_00_01).
Bottom line, I don't think larsoft v06 is ready, and microboone doesn't sign off.
#3 Updated by Marc Paterno over 3 years ago
- File root-5-6-difference-by-event.pdf root-5-6-difference-by-event.pdf added
- File root-5-6-rss-usage.pdf root-5-6-rss-usage.pdf added
We have run several jobs, one using ROOT5 and one using ROOT6, reading one the files provided by Herb. In our jobs we ran only ROOT input and ROOT output (not using any MicroBooNE modules), to look at how ROOT i/o may have changed. We have turned off fast cloning, because that is how the real jobs were run.
Attached are two plots. One shows no event-by-event memory usage increase either in ROOT 5 or ROOT 6. The other shows that the difference in memory (RSS) usage between ROOT5 and ROOT 6 is on the order of 200MB.
Despite not being relevant for grid memory limits, we also looked at Vsize measurements, and they also show no increase event-by-event.
We are currently attempting to run the full job, including all the MicroBooNE modules and running over all the data files, to look at the memory tracker output.
#4 Updated by Lynn Garren over 3 years ago
Thanks to Paul Russo, the bug is fixed in root v6_06_04b. A secondary workaround was also made in art itself.
art v2_00_03, nutools v2_00_01, and larsoftobj v1_02_02 are now available with the new releases of art, canvas, and gallery. larsoft v06_00_00_rc5 is tagged and building.