Project

General

Profile

Bug #20427

Corrupt file in CVMFS uboonedata produces abnormal memory usage

Added by Lorena Escudero sanchez over 1 year ago. Updated over 1 year ago.

Status:
Closed
Priority:
Normal
Assignee:
-
Target version:
-
Start date:
07/24/2018
Due date:
% Done:

100%

Estimated time:
Duration:

Description

Recently it was brought to our attention that abnormal memory usage occurred when running certain configurations of Pandora via uboonecode v06_73_00. We discarded any memory leaks and/or “invalid writes” by several of our usual tests (see for example valgrind output here [[https://www.hep.phy.cam.ac.uk/~escudero/valgrind_v06_73_00_myreco_uboone_data_mcc8_driver_stage2.txt]]). However, the virtual and resident size of the process was indeed getting huge (>30GB). After some investigation, I found out that the problem actually arises when tiny XML tries to parse a corrupted XML file from the uboonedata installation on CVMFS. It looks like the relevant file PandoraSvm_v03_11_00.xml is fine in the repository, but was truncated in the installation. In our experience, tiny XML normally reports an invalid XML file when this kind of thing happens, but this time somehow gets stuck in a loop, allocating memory. We are not looking to modify tiny XML ourselves, and have never had any issues with it in the past, but one thing that can be done is to fix the corrupted xml file and understand whether this kind of problem can be mitigated in future. Further details below:

Corrupted file:
/cvmfs/uboone.opensciencegrid.org/products/uboonedata/v01_18_09/PandoraData/PandoraSvm_v03_11_00.xml
(its size is 8.5 MB vs the 22MB it should be as the previous v01_18_08 and next v01_18_10 versions): if you open the file it is cut in a not-ending line, and does not contain relevant SVM models Pandora needs.

This happens to the installation on CVMFS, its fine in the repository: if you get a local copy of uboonedata with version v01_18_09 the file is fine and the process runs without problems..
Please replace the copy of PandoraSvm_v03_11_00.xml in /cvmfs/uboone.opensciencegrid.org/products/uboonedata/v01_18_09/PandoraData/PandoraSvm_v03_11_00.xml

History

#1 Updated by Lynn Garren over 1 year ago

  • Project changed from LArSoft to uBooNE code
  • Occurs In deleted (v06_73_00)
  • Experiment deleted (MicroBooNE)

This is a MicroBooNE problem

#2 Updated by Lynn Garren over 1 year ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

Fixed by the relevant parties.



Also available in: Atom PDF