Corrupt file in CVMFS uboonedata produces abnormal memory usage
Recently it was brought to our attention that abnormal memory usage occurred when running certain configurations of Pandora via uboonecode v06_73_00. We discarded any memory leaks and/or “invalid writes” by several of our usual tests (see for example valgrind output here [[https://www.hep.phy.cam.ac.uk/~escudero/valgrind_v06_73_00_myreco_uboone_data_mcc8_driver_stage2.txt]]). However, the virtual and resident size of the process was indeed getting huge (>30GB). After some investigation, I found out that the problem actually arises when tiny XML tries to parse a corrupted XML file from the uboonedata installation on CVMFS. It looks like the relevant file PandoraSvm_v03_11_00.xml is fine in the repository, but was truncated in the installation. In our experience, tiny XML normally reports an invalid XML file when this kind of thing happens, but this time somehow gets stuck in a loop, allocating memory. We are not looking to modify tiny XML ourselves, and have never had any issues with it in the past, but one thing that can be done is to fix the corrupted xml file and understand whether this kind of problem can be mitigated in future. Further details below:
(its size is 8.5 MB vs the 22MB it should be as the previous v01_18_08 and next v01_18_10 versions): if you open the file it is cut in a not-ending line, and does not contain relevant SVM models Pandora needs.
This happens to the installation on CVMFS, its fine in the repository: if you get a local copy of uboonedata with version v01_18_09 the file is fine and the process runs without problems..
Please replace the copy of PandoraSvm_v03_11_00.xml in /cvmfs/uboone.opensciencegrid.org/products/uboonedata/v01_18_09/PandoraData/PandoraSvm_v03_11_00.xml