Project

General

Profile

Support #5655

Excessive memory use during file merge

Added by Andrei Gaponenko over 5 years ago. Updated almost 2 years ago.

Status:
Closed
Priority:
Normal
Category:
I/O
Target version:
-
Start date:
03/13/2014
Due date:
% Done:

0%

Estimated time:
Scope:
Internal
Experiment:
Mu2e
SSI Package:
art
Duration:

Description

Hello,

Running

mu2e -c fileMerge.fcl /mu2e/data/tdr/beam/g4s4p4f1/tdr.beam.g4s4f.flash.1025a_1025a_1316a_1316a_1316b.14325221/good/*/detectorBeamFlash.root

requires 6 GB of resident memory, which is quite excessive,
given that the total size of inputs is 5 * 263 MB.

The fcl file is pasted below
Andrei

process_name: fileMerge
source: { module_type: RootInput }
physics: { e1: [out] end_paths: [e1] }
outputs: { out: { module_type: RootOutput fileName: "dataMerged.root" } }
services.scheduler.fileMode: MERGE

History

#1 Updated by Christopher Green over 5 years ago

  • Tracker changed from Bug to Support
  • Category set to I/O
  • Status changed from New to Feedback
  • Assignee set to Christopher Green
  • Experiment Mu2e added
  • Experiment deleted (-)
  • SSI Package art added
  • SSI Package deleted ()

Despite the unhelpful name, we think "NOMERGE" is the mode you want here. You must however guarantee that you never return to either a run or subrun after starting to process another run or subrun.

Let us know if this works for you.

#2 Updated by Christopher Green about 5 years ago

Is this still an issue or can it be resolved / closed?

#3 Updated by Andrei Gaponenko about 5 years ago

Hi,

Is this still an issue or can it be resolved / closed?

This may be addressed, along with #5217, by resolving #5805.
Per #5805 we need art v1.11.00 to be able to test the fix.
Mu2e uses v1.10.00b at the moment, so I can't give an answer right now.

Andrei

#4 Updated by Kyle Knoepfel over 4 years ago

  • Target version set to 521

#5 Updated by Kyle Knoepfel about 3 years ago

  • Status changed from Feedback to Closed

Closing per author request.

#6 Updated by Kyle Knoepfel almost 2 years ago

  • Target version deleted (521)


Also available in: Atom PDF