Project

General

Profile

Bug #1328

reco_near_spill_data_base_dogwood5.C job crashing

Added by Arthur Kreymer over 8 years ago. Updated over 8 years ago.

Status:
New
Priority:
Normal
Assignee:
Start date:
05/26/2011
Due date:
% Done:

30%

Estimated time:
Duration:

Description

Unresolved. Initial bugtrack entry by George Irwin (16May2011)

Rashid reported a job crashing with
S11-04-29-R2-06, built with GCC 4.5.1 on SLF5 linux machines.
Initial suspicion was that this problem was unique to SLF5 and perhaps to GCC 4.5.1.
The results so far disprove the suspicion.

The job was launched with the command:

loon -b -q ../asciidb/set_tsql_override.C \
reco_near_spill_data_base_dogwood5.C \
N00017725_0001.mdaq.root

See details at

http://www-numi.fnal.gov/offline_software/srt_public_context/WebDocs/bug_tracking/bg0009.html

History

#1 Updated by Arthur Kreymer over 8 years ago

  • Assignee set to George Irwin

Here is the Diagnostic information from the Bug Tracking page, for the record

Below is a summary of running this job with
several Linux and and GCC configurations.
In addition, the job was run on an Intel Mac
with the SnowLeopard OS. All jobs crashed
on run 17725_0001. The Root version for the
development jobs was 5.29/03. The snapshot
release S11-04-29-R2-06 had been built
against Root 5.29/02.

The following jobs ("A") crashed at the same
point in the code on the same spill (103898):

minos51 (SLF4) with GCC 4.5.1 (development)
minos53 (SLF5) with GCC 4.5.1 (development and
S11-04-29-R2-06)
minos53 (SLF5) with GCC 3.4.3 (development)
SLAC (SLF5) with GCC 4.1.2 (development)

Another job ("B") crashed at a different point
in the code, but on the same spill (103898):

minos51 (SLF4) with GCC 3.4.3 (development)

The SnowLeopard job ("C") crashed at another
point of code on a later spill (108092):

Mac 10.6.7 with GCC 4.2.1 (development)

*The "A" jobs all ended on spill 103898
with similar messages and GDB backtraces: *
Program received signal SIGSEGV, Segmentation fault.
0x005c8175 in _int_free () from /lib/libc.so.6
(gdb) bt
#0 0x005c8175 in _int_free () from /lib/libc.so.6
#1 0x005c8b09 in free () from /lib/libc.so.6
#2 0xf7fd24a5 in operator delete (ptr=0xec03)
at ../../../../../gcc-3.4.3/libstdc++-v3/libsupc++/del_op.cc:40
#3 0xf3c087e3 in TStorage::ReAllocChar (ovp=0xeb6cda8 "", size=2208,
oldsize=1108)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/base/src/TStorage.cxx:280
#4 0xf3bb35fc in TBuffer::Expand (this=0xfe85240, newsize=2200)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/base/src/TBuffer.cxx:189
#5 0xf4439e92 in TBufferFile::WriteFloat (this=0xfe85240, f=29468)
at include/TBufferFile.h:401
#6 0xf7ed3974 in operator<< (buf=..., f=29468)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/include/TBuffer.h:349
#7 0xf4527fc9 in TStreamerInfo::WriteBufferAux (this=0xdf47aa0,
b=..., arr=@0xfffc98e0, first=1, narr=53, eoffset=64, arrayMode=1)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/io/io/src/TStreamerInfoWriteBuffer.cxx:233
#8 0xf451bea2 in TStreamerInfo::WriteBufferClones (this=0xdf47aa0, b=...,
clones=0xde24168, nc=53, first=1, eoffset=64)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/io/io/src/TStreamerInfoWriteBuffer.cxx:900
#9 0xf4c71e01 in TBranchElement::FillLeaves (this=0xdfe9aa8, b=...)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/tree/tree/src/TBranchElement.cxx:1400
#10 0xf4c63f87 in TBranch::Fill (this=0xdfe9aa8)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/tree/tree/src/TBranch.cxx:893
#11 0xf4c71153 in TBranchElement::Fill (this=0xdfe9aa8)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/tree/tree/src/TBranchElement.cxx:1176
#12 0xf4c712c1 in TBranchElement::Fill (this=0xdfda390)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/tree/tree/src/TBranchElement.cxx:1201
#13 0xf4c712c1 in TBranchElement::Fill (this=0xdf54c18)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/tree/tree/src/TBranchElement.cxx:1201
#14 0xf4cd1944 in TTree::Fill (this=0xdefd3e0)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/tree/tree/src/TTree.cxx:3918
#15 0xf5f58beb in PerOutputStream::Store (this=0x8ae0ac8)
at /grid/fermiapp/minos/minossoft/packages/Persistency/HEAD/PerOutputStream.cxx:624
#16 0xf5f5a648 in PerOutputStreamManager::Put (this=0x89cfbc0, mom=0x89ad400)
at /grid/fermiapp/minos/minossoft/packages/Persistency/HEAD/PerOutputStreamManager.cxx:198
#17 0xf630593e in IoOutputModule::Put (this=0x8ae0150, mom=0x89ad400)
at /grid/fermiapp/minos/minossoft/packages/IoModules/HEAD/IoOutputModule.cxx:194

*The "B" job also ended on spill 103898, but
with a different message and GDB backtrace: *

  • glibc detected * double free or corruption (!prev): 0x18f969c8 *

Program received signal SIGABRT, Aborted.
[Switching to Thread 4044646624 (LWP 11148)]
0xffffe410 in __kernel_vsyscall ()
(gdb) bt
#0 0xffffe410 in __kernel_vsyscall ()
#1 0x00bbb825 in raise () from /lib/tls/libc.so.6
#2 0x00bbd289 in abort () from /lib/tls/libc.so.6
#3 0x00befcda in __libc_message () from /lib/tls/libc.so.6
#4 0x00bf656f in _int_free () from /lib/tls/libc.so.6
#5 0x00bf694a in free () from /lib/tls/libc.so.6
#6 0xf7fd24a5 in operator delete (ptr=0x0)
at ../../../../../gcc-3.4.3/libstdc++-v3/libsupc++/del_op.cc:40
#7 0xf3c09ac4 in TStorage::ObjectDealloc (vp=0x18f969c8)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/base/src/TStorage.cxx:358
#8 0xf3be06f6 in TObject::operator delete (ptr=0x18f969c8)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/base/src/TObject.cxx:1003
#9 0xf7803206 in ~CandDigit (this=0x18f969c8)
at /grid/fermiapp/minos/minossoft/packages/CandDigit/HEAD/CandDigit.cxx:137
#10 0xf6c3ef15 in CandRefer::DropRefer (this=0x18f96b34)
at /grid/fermiapp/minos/minossoft/packages/Candidate/HEAD/CandRefer.cxx:92
#11 0xf6c3ed9f in CandRefer::~CandRefer ()
at /grid/fermiapp/minos/products/prd/gcc/v3_4_3/Linux+2.6-2.3.4/bin/../lib/gcc/i686-pc-linux-gnu/3.4.3/../../../../include/c++/3.4.3/bits/locale_facets.tcc:2444
#12 0xf6c3b67b in ~CandHandle (this=0x18f96b28)
at /grid/fermiapp/minos/minossoft/packages/Candidate/HEAD/CandHandle.cxx:75
#13 0xf780512b in CandDigitHandle::~CandDigitHandle$delete ()
at /grid/fermiapp/minos/products/prd/gcc/v3_4_3/Linux+2.6-2.3.4/bin/../lib/gcc/i686-pc-linux-gnu/3.4.3/../../../../include/c++/3.4.3/bits/locale_facets.tcc:2444
#14 0xf3c5ab8b in TCollection::GarbageCollect (obj=0x18f96b28)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/cont/src/TCollection.cxx:591
#15 0xf3c6577e in TObjArray::Delete (this=0x188eeea8)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/cont/src/TObjArray.cxx:335
#16 0xf6c356b9 in ~CandBase (this=0x188eee60)
at /grid/fermiapp/minos/minossoft/packages/Candidate/HEAD/CandBase.cxx:97
#17 0xf7808540 in CandDigitList::~CandDigitList$delete ()
at /grid/fermiapp/minos/products/prd/gcc/v3_4_3/Linux+2.6-2.3.4/bin/../lib/gcc/i686-pc-linux-gnu/3.4.3/../../../../include/c++/3.4.3/bits/locale_facets.tcc:2444
#18 0xf6c3ef15 in CandRefer::DropRefer (this=0x18fbbadc)
at /grid/fermiapp/minos/minossoft/packages/Candidate/HEAD/CandRefer.cxx:92
#19 0xf6c3ed9f in CandRefer::~CandRefer ()
at /grid/fermiapp/minos/products/prd/gcc/v3_4_3/Linux+2.6-2.3.4/bin/../lib/gcc/i686-pc-linux-gnu/3.4.3/../../../../include/c++/3.4.3/bits/locale_facets.tcc:2444
#20 0xf6c3b67b in ~CandHandle (this=0x18fbbad0)
at /grid/fermiapp/minos/minossoft/packages/Candidate/HEAD/CandHandle.cxx:75
#21 0xf78094bd in ~CandDigitListHandle (this=0x18fbbad0)
at /grid/fermiapp/minos/minossoft/packages/CandDigit/HEAD/CandDigitListHandle.cxx:47
#22 0xf3c5ab8b in TCollection::GarbageCollect (obj=0x18fbbad0)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/cont/src/TCollection.cxx:591
#23 0xf3c6577e in TObjArray::Delete (this=0x18e6528c)
at /grid/fermiapp/minos/products/prd/MINOS_ROOT/Linux2.4-GCC_3_4/trunk/core/cont/src/TObjArray.cxx:335
#24 0xf5ea8f9f in ~RecMinos (this=0x18e65208)
at /grid/fermiapp/minos/minossoft/packages/Record/HEAD/RecMinos.cxx:44
#25 0xf6c90ddb in CandRecord::~CandRecord$delete ()
at /grid/fermiapp/minos/products/prd/gcc/v3_4_3/Linux+2.6-2.3.4/bin/../lib/gcc/i686-pc-linux-gnu/3.4.3/../../../../include/c++/3.4.3/bits/locale_facets.tcc:2444
#26 0xf5cc5d6c in MomNavigator::Clear (this=0x89aad88, opt=0xf5d65506 "")
at /grid/fermiapp/minos/minossoft/packages/MinosObjectMap/HEAD/MomNavigator.cxx:327
#27 0xf5d2c2be in JobCRecord::Reset (this=0xfffca050)
at /grid/fermiapp/minos/minossoft/packages/JobControl/HEAD/JobCRecord.cxx:87

*The "C" job ended on spill 108092
with this message and GDB backtrace: *

Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_PROTECTION_FAILURE at address: 0x000003f0
0x07448856 in NavSet::Fill (this=0x2cf8caf0, source=0x2f2f44d8, classname=0x5ab33bc "CandStripHandle", offset=0) at /minosbase/minossoft/packages/Navigation/HEAD/NavSet.cxx:552
552 if ( ok && ( suppressClassCheck || sourceObj->InheritsFrom(classname) )
(gdb) bt
#0 0x07448856 in NavSet::Fill (this=0x2cf8caf0, source=0x2f2f44d8, classname=0x5ab33bc "CandStripHandle", offset=0) at /minosbase/minossoft/packages/Navigation/HEAD/NavSet.cxx:552
#1 0x07448b95 in NavSet::NavSet (this=0x2cf8caf0, source=0x2f2f44d8, classname=0x5ab33bc "CandStripHandle", offset=0) at /minosbase/minossoft/packages/Navigation/HEAD/NavSet.cxx:969
#2 0x0744224c in NavItr::NavItr (this=0xbffccf2c, sourceItr=0xbffccf60, classname=0x5ab33bc "CandStripHandle", offset=0) at /minosbase/minossoft/packages/Navigation/HEAD/NavItr.cxx:551
#3 0x059d869a in CandStripHandleItr::CandStripHandleItr (this=0xbffccf2c, sourceItr=@0xbffccf60) at /minosbase/minossoft/packages/RecoBase/HEAD/CandStripHandle.cxx:347
#4 0x059bdb3b in CandRecoHandle::GetPlaneCharge (this=0x2f217e60, iplane=1, caltype_t=CalStripType::kPE) at /minosbase/minossoft/packages/RecoBase/HEAD/CandRecoHandle.cxx:337
#5 0x0e1251e3 in GetMaxTrackLikePlane (event=0x2f217e60, track=0x2f21a0a0, n=6) at /minosbase/minossoft/packages/MuonRemoval/HEAD/SelectEvent.cxx:133
#6 0x0e0eeaef in AlgMuonRemoval::FillTrkInfo (this=0x235a0400, event=0x2f217e60, track=0x2f21a0a0) at /minosbase/minossoft/packages/MuonRemoval/HEAD/AlgMuonRemoval.cxx:316
#7 0x0e0ef7b0 in AlgMuonRemoval::RunAlg (this=0x235a0400, ac=@0x1f1cf330, ch=@0xbffcd580, cx=@0xbffcd680) at /minosbase/minossoft/packages/MuonRemoval/HEAD/AlgMuonRemoval.cxx:132
#8 0x05d8deff in AlgHandle::RunAlg (this=0xbffcd604, ch=@0xbffcd580, cx=@0xbffcd680) at /minosbase/minossoft/packages/Algorithm/HEAD/AlgHandle.cxx:68
#9 0x04a99302 in CandDigitList::CandDigitList (this=0x2ce7afb0, ah=@0xbffcd604, ch=@0xbffcd580, cx=@0xbffcd680) at /minosbase/minossoft/packages/CandDigit/HEAD/CandDigitList.cxx:79
#10 0x04a9936d in CandDigitList::MakeCandidate (ah=@0xbffcd604, cx=@0xbffcd680) at /minosbase/minossoft/packages/CandDigit/HEAD/CandDigitList.cxx:154
#11 0x0e1229bc in RemoveMuon::Reco (this=0xedd85a0, mom=0xbfc4050) at /minosbase/minossoft/packages/MuonRemoval/HEAD/RemoveMuon.cxx:244

#2 Updated by Arthur Kreymer over 8 years ago

From George Irwin :

I failed to put this result in redmine.
It rejected the entry after I clicked "submit."
There's a second one to follow for Snow Leopard.

Here is a segment of valgrind output for a
job on SLAC node noric01 running RHEL 5.6.
Because valgrind is so slow, and I wanted
to get to spill 103898, where valgrind usually
signals (on both RHEL 5.6 and Snow Leopard),
I skipped the first 3800 spills on the RHEL 5.6
job. Unfortunately, this changes the spill on
which valgrind trips. However, the type of
error is typical of several of the jobs which
have failed. I'm running minossoft and root
development updated within the last week. I
have the AlgFitTrackCam message level set to
Debug to print some helpful stuff.

noric01>valgrind --tool=memcheck --db-attach=yes \
--gen-suppressions=yes --suppressions=$ROOTSYS/etc/valgrind-root.supp \
--error-limit=no --leak-check=full --show-reachable=yes --dsymutil=yes \
-v loon -bq ../asciidb/set_tsql_override.C \
"reco_near_spill_data_base_dogwood5.C(3800,10000000)" \
../../../root_data/N00017725_0001.mdaq.root

...
=D= AlgFitTr [17725|103849] AlgFitTrackCam.cxx:2207> Checking Last Plane:
Q/P=-0.493752 MajorityCurvature = -0.567568
StoreData [256] x_k4 = -0.493752 Entry=3
GoForwards, Finished
StoreData [256] x_k4 = -0.493752 Entry=4
GoBackwards, carry out fit in negative z direction
Fitting from StartPlane=256 to EndPlane=95 end of range plane = 256
StoreData [246] x_k4 = -0.493711 Entry=4
StoreData [236] x_k4 = -0.493664 Entry=4
StoreData [226] x_k4 = -0.46916 Entry=4
StoreData [206] x_k4 = -0.342215 Entry=4
StoreData [191] x_k4 = -0.288534 Entry=4
StoreData [186] x_k4 = -0.225446 Entry=4
StoreData [176] x_k4 = -0.117335 Entry=4
StoreData [171] x_k4 = 0.0328455 Entry=4
StoreData [166] x_k4 = -0.0341547 Entry=4
StoreData [161] x_k4 = -0.27568 Entry=4
StoreData [156] x_k4 = -0.18371 Entry=4
StoreData [151] x_k4 = -0.252193 Entry=4
StoreData [146] x_k4 = -0.151485 Entry=4
StoreData [141] x_k4 = -0.232328 Entry=4
StoreData [136] x_k4 = -0.132019 Entry=4
StoreData [131] x_k4 = -0.150647 Entry=4
StoreData [119] x_k4 = -0.126698 Entry=5
StoreData [117] x_k4 = -0.111808 Entry=5
StoreData [116] x_k4 = -0.0903476 Entry=5
StoreData [115] x_k4 = -0.0884474 Entry=5
StoreData [114] x_k4 = -0.0876374 Entry=5
StoreData [113] x_k4 = -0.0883492 Entry=5
StoreData [111] x_k4 = -0.0910875 Entry=5
StoreData [109] x_k4 = -0.0917177 Entry=5
StoreData [108] x_k4 = -0.0849313 Entry=5
StoreData [107] x_k4 = -0.0817926 Entry=5
StoreData [106] x_k4 = -0.0877233 Entry=5
StoreData [105] x_k4 = -0.083761 Entry=5
StoreData [104] x_k4 = -0.087858 Entry=5
StoreData [103] x_k4 = -0.0877399 Entry=5
StoreData [102] x_k4 = -0.0902966 Entry=5
StoreData [101] x_k4 = -0.0963215 Entry=5
StoreData [100] x_k4 = -0.0972431 Entry=5
StoreData [99] x_k4 = -0.111836 Entry=5
StoreData [96] x_k4 = -0.122059 Entry=5
StoreData [95] x_k4 = -0.131872 Entry=5
GoBackwards, Finished
Set Track Properties:

Q/P = -0.11329, Q/P(biased) = -0.168728, sigma(Q/P) = 0.0194026 Sigma Q/P
_biased 0.0236878
SetRangeAnddS from swimmer values between 99/256
1587 Invalid read of size 4
1587 at 0x535A69C: CandRefer::operator()() const (CandRefer.h:51)
1587 by 0x53580CB: CandHandle::GetCandBase() const
(CandHandle.cxx:243)
1587 by 0x53580FC: CandHandle::operator==(CandHandle const&) const
(CandHandle.cxx:111)
1587 by 0x5358AC3: CandHandle::FindDaughter(CandHandle const*) const
(CandHandle.cxx:181)
1587 by 0x535914D: CandHandle::RemoveDaughter(CandHandle*)
(CandHandle.cxx:421)
1587 by 0xF82B393: AlgFitTrackCam::CleanNDLists(CandFitTrackHandle&,
CandContext&) (AlgFitTrackCam.cxx:3772)
1587 by 0xF83CCF4: AlgFitTrackCam::RunAlg(AlgConfig&, CandHandle&,
CandContext&) (AlgFitTrackCam.cxx:229)
1587 by 0x538E1BA: AlgHandle::RunAlg(CandHandle&, CandContext&)
(AlgHandle.cxx:68)
1587 by 0xF849923: CandFitTrackCam::CandFitTrackCam(AlgHandle&,
CandHandle&, CandContext&) (CandFitTrackCam.cxx:37)
1587 by 0xF84999A: CandFitTrackCam::MakeCandidate(AlgHandle&,
CandContext&) (CandFitTrackCam.cxx:69)
1587 by 0xF848CE0: AlgFitTrackCamList::RunAlg(AlgConfig&,
CandHandle&, CandContext&) (AlgFitTrackCamList.cxx:150)
1587 by 0x538E1BA: AlgHandle::RunAlg(CandHandle&, CandContext&)
(AlgHandle.cxx:68)
1587 Address 0x200b5d30 is 16 bytes inside a block of size 32 free'd
1587 at 0x4022234: operator delete(void*) (vg_replace_malloc.c:346)
1587 by 0x8A0E796: TStorage::ObjectDealloc(void*) (TStorage.cxx:358)
1587 by 0x89E35A3: TObject::operator delete(void*) (TObject.cxx:1003)
1587 by 0x46D75C8: CandDigitHandle::~CandDigitHandle()
(CandDigitHandle.cxx:48)
1587 by 0x53520A9: CandBase::RemoveDaughter(CandHandle*)
(CandBase.cxx:289)
1587 by 0x5359172: CandHandle::RemoveDaughter(CandHandle*)

(CandHandle.cxx:422)
1587 by 0xF82B393: AlgFitTrackCam::CleanNDLists(CandFitTrackHandle&,
CandContext&) (AlgFitTrackCam.cxx:3772)
1587 by 0xF83CCF4: AlgFitTrackCam::RunAlg(AlgConfig&, CandHandle&,
CandContext&) (AlgFitTrackCam.cxx:229)
1587 by 0x538E1BA: AlgHandle::RunAlg(CandHandle&, CandContext&)
(AlgHandle.cxx:68)
1587 by 0xF849923: CandFitTrackCam::CandFitTrackCam(AlgHandle&,
CandHandle&, CandContext&) (CandFitTrackCam.cxx:37)
1587 by 0xF84999A: CandFitTrackCam::MakeCandidate(AlgHandle&,
CandContext&) (CandFitTrackCam.cxx:69)
1587 by 0xF848CE0: AlgFitTrackCamList::RunAlg(AlgConfig&,
CandHandle&, CandContext&) (AlgFitTrackCamList.cxx:150)
1587
1587
1587 ---- Attach to debugger ? --- [Return/N/n/Y/y/C/c] ---- c
1587
1587 ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ---- c
RunAlg, New track, Run: 17725 Snarl: 103849
=D= AlgFitTr [17725|103849] AlgFitTrackCam.cxx:3304> GenerateNDSpectStrips
RunTheFitter, Call methods in the appropriate order
StoreData [6] x_k4 = 0 Entry=1
GoForwards, carry out fit in positive z direction
Fitting from StartPlane=6 to EndPlane=61
StoreData [11] x_k4 = 2.2517e-05 Entry=1
StoreData [16] x_k4 = 3.57451e-05 Entry=1
StoreData [26] x_k4 = -0.00393066 Entry=1
StoreData [31] x_k4 = -0.00380544 Entry=1
StoreData [41] x_k4 = -0.739604 Entry=1
StoreData [46] x_k4 = -0.603017 Entry=1
StoreData [51] x_k4 = -0.490573 Entry=1

=D= AlgFitTr [17725|103849] AlgFitTrackCam.cxx:2207> Checking Last Plane:
Q/P=-0.547418 MajorityCurvature = -0.375
StoreData [61] x_k4 = -0.547418 Entry=1
GoForwards, Finished
=D= AlgFitTr [17725|103849] AlgFitTrackCam.cxx:305> ShowerStrips, Look for
large vertex shower

...

#3 Updated by Arthur Kreymer over 8 years ago

Date: Tue, 14 Jun 2011 22:51:01 -0700 (PDT)
From: George M. Irwin <>
To: Arthur Kreymer <>, Robert Hatcher <>
Subject: valgrind result for Snow Leopard

removed this partial output, will post the complete output

#4 Updated by Robert Hatcher over 8 years ago

content removed

release                         root              os-build  os-run  works?
S11-04-29-R2-06   v5-29-02   SLF4          SLF4       no

#5 Updated by George Irwin over 8 years ago

This is a segment of valgrind output for a
job on my PowerMac laptop running Snow Leopard.

Valgrind usually finds problems for spill 103898
on both Mac and Linux.  On the Mac, valgrind itself
crashes on this spill.

There are several valgrind signals in the earlier
snarls.  These error types occur in various other
jobs that I've tried, on both MacOSX and Linux.

I'm running minossoft and root development updated
within the last week.  I have the AlgFitTrackCam
message level set to Debug for additional print.

Leopard> valgrind --tool=memcheck --db-attach=yes \
--gen-suppressions=yes --suppressions=$ROOTSYS/etc/valgrind-root.supp \
--error-limit=no --leak-check=full --show-reachable=yes --dsymutil=yes \
-v loon -bq ../asciidb/set_tsql_override.C \
reco_near_spill_data_base_dogwood5.C \
../../../root_data/N00017725_0001.mdaq.root

...

RunAlg, New track, Run: 17725 Snarl: 99986
=D= AlgFitTr [17725|99986] AlgFitTrackCam.cxx:3304> GenerateNDSpectStrips
RunTheFitter, Call methods in the appropriate order
 StoreData [1]  x_k[4] = 0 Entry=1
GoForwards, carry out fit in positive z direction
Fitting from StartPlane=1 to EndPlane=96
LoadMinosPDG: initializing PDG database (minos version)
LoadMinosPDG: reading /minosbase/minossoft/releases/development/etc/minos_pdg_table.txt
MCApplication::InitMC initializing MC with configuration script /minosbase/minossoft/releases/development/macros/PTSim_g3Config.C:
==28682== Syscall param wait4(pid) contains uninitialised byte(s)
==28682==    at 0x2E514FA: wait4 (in /usr/lib/libSystem.B.dylib)
==28682==    by 0x286E964: G__process_cmd (pause.cxx:2676)
==28682==    by 0x1BBCBD7: TCint::ProcessLine(char const*, TInterpreter::EErrorCode*) (TCint.cxx:522)
==28682==    by 0x1CC6631: TCint::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) (TCint.cxx:601)
==28682==    by 0xE4EA80F: MCApplication::InitMC(char const*, VldContext const&, char const*) (MCApplication.cxx:145)
==28682==    by 0x107CA7AA: GeoSwimmer::Initialize(VldContext const&) (GeoSwimmer.cxx:95)
==28682==    by 0x118B11D5: AlgFitTrackCam::Swim(double*, double*, int, int, bool, double*, double*, double*) (AlgFitTrackCam.cxx:1524)
==28682==    by 0x118B6B9A: AlgFitTrackCam::GetCombiPropagator(int, int, bool) (AlgFitTrackCam.cxx:1465)
==28682==    by 0x118B919A: AlgFitTrackCam::GoForwards() (AlgFitTrackCam.cxx:1292)
==28682==    by 0x118BA0D1: AlgFitTrackCam::RunTheFitter(CandFitTrackCamHandle&) (AlgFitTrackCam.cxx:417)
==28682==    by 0x118BBA04: AlgFitTrackCam::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgFitTrackCam.cxx:223)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682== 
==28682== 
==28682== ---- Attach to debugger ? --- [Return/N/n/Y/y/C/c] ---- c
==28682== 
==28682== ---- Print suppression ? --- [Return/N/n/Y/y/C/c] ---- n

...

=D= AlgFitTr [17725|100030] AlgFitTrackCam.cxx:2207>  Checking Last Plane: Q/P=-0.17926894 MajorityCurvature = -1
 StoreData [281]  x_k[4] = -0.179269 Entry=3
GoForwards, Finished
 StoreData [281]  x_k[4] = -0.179269 Entry=4
GoBackwards, carry out fit in negative z direction
Fitting from StartPlane=281 to EndPlane=84 end of range plane = 281

...

 StoreData [100]  x_k[4] = -0.0792361 Entry=5
 StoreData [99]  x_k[4] = -0.0791329 Entry=5
 StoreData [98]  x_k[4] = -0.078794 Entry=5
 StoreData [97]  x_k[4] = -0.0779377 Entry=5
 StoreData [96]  x_k[4] = -0.0773916 Entry=5
 StoreData [95]  x_k[4] = -0.0771653 Entry=5
 StoreData [94]  x_k[4] = -0.0765453 Entry=5
 StoreData [93]  x_k[4] = -0.0773027 Entry=5
 StoreData [92]  x_k[4] = -0.0770197 Entry=5
 StoreData [91]  x_k[4] = -0.077254 Entry=5
 StoreData [90]  x_k[4] = -0.0766702 Entry=5
 StoreData [89]  x_k[4] = -0.0765587 Entry=5
 StoreData [88]  x_k[4] = -0.0762214 Entry=5
 StoreData [87]  x_k[4] = -0.0756136 Entry=5
 StoreData [84]  x_k[4] = -0.0734872 Entry=5
GoBackwards, Finished
Set Track Properties:
 Q/P = -0.0765966, Q/P(biased) = -0.0753971, sigma(Q/P) = 0.00749159 Sigma Q/P _biased 0.00776618
SetRangeAnddS from swimmer values between 87/281
==28682== Invalid read of size 4
==28682==    at 0x5FC229D: CandRefer::operator()() const (CandRefer.h:51)
==28682==    by 0x5FBFE0D: CandHandle::GetCandBase() const (CandHandle.cxx:243)
==28682==    by 0x5FBFE2E: CandHandle::operator==(CandHandle const&) const (CandHandle.cxx:111)
==28682==    by 0x5FC0730: CandHandle::FindDaughter(CandHandle const*) const (CandHandle.cxx:181)
==28682==    by 0x5FC0D53: CandHandle::RemoveDaughter(CandHandle*) (CandHandle.cxx:421)
==28682==    by 0x118A8277: AlgFitTrackCam::CleanNDLists(CandFitTrackHandle&, CandContext&) (AlgFitTrackCam.cxx:3772)
==28682==    by 0x118BBA38: AlgFitTrackCam::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgFitTrackCam.cxx:229)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x118C5BC8: CandFitTrackCam::CandFitTrackCam(AlgHandle&, CandHandle&, CandContext&) (CandFitTrackCam.cxx:37)
==28682==    by 0x118C5C34: CandFitTrackCam::MakeCandidate(AlgHandle&, CandContext&) (CandFitTrackCam.cxx:69)
==28682==    by 0x118C4F7A: AlgFitTrackCamList::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgFitTrackCamList.cxx:150)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==  Address 0x32aae4e0 is 16 bytes inside a block of size 32 free'd
==28682==    at 0x30BB7: operator delete(void*) (vg_replace_malloc.c:346)
==28682==    by 0x1BAF45F: TStorage::ObjectDealloc(void*) (TStorage.cxx:358)
==28682==    by 0x1BA3547: TObject::operator delete(void*) (TObject.cxx:1003)
==28682==    by 0x4D25F6D: CandDigitHandle::~CandDigitHandle() (CandDigitHandle.cxx:48)
==28682==    by 0x5FBAA22: CandBase::RemoveDaughter(CandHandle*) (CandBase.cxx:289)
==28682==    by 0x5FC0D78: CandHandle::RemoveDaughter(CandHandle*) (CandHandle.cxx:422)
==28682==    by 0x118A8277: AlgFitTrackCam::CleanNDLists(CandFitTrackHandle&, CandContext&) (AlgFitTrackCam.cxx:3772)
==28682==    by 0x118BBA38: AlgFitTrackCam::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgFitTrackCam.cxx:229)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x118C5BC8: CandFitTrackCam::CandFitTrackCam(AlgHandle&, CandHandle&, CandContext&) (CandFitTrackCam.cxx:37)
==28682==    by 0x118C5C34: CandFitTrackCam::MakeCandidate(AlgHandle&, CandContext&) (CandFitTrackCam.cxx:69)
==28682==    by 0x118C4F7A: AlgFitTrackCamList::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgFitTrackCamList.cxx:150)
==28682== 
RunAlg, New track, Run: 17725 Snarl: 100030

...

Fitting from StartPlane=117 to EndPlane=96 end of range plane = 112
 StoreData [111]  x_k[4] = -3.51056 Entry=5
 StoreData [106]  x_k[4] = -2.17477 Entry=5
 StoreData [101]  x_k[4] = -1.53365 Entry=5
 StoreData [98]  x_k[4] = -1.30297 Entry=5
 StoreData [96]  x_k[4] = -1.16014 Entry=5
GoBackwards, Finished
Set Track Properties:
 Q/P = -1.17522, Q/P(biased) = -1.18344, sigma(Q/P) = 0.289376 Sigma Q/P _biased 0.166934
SetRangeAnddS from swimmer values between 96/117
==28682== Conditional jump or move depends on uninitialised value(s)
==28682==    at 0x829BECB: deflateEnd (in /usr/lib/libz.1.2.3.dylib)
==28682==    by 0x829CDE7: deflateSetDictionary (in /usr/lib/libz.1.2.3.dylib)
==28682==    by 0x829DADB: deflate (in /usr/lib/libz.1.2.3.dylib)
==28682==    by 0x1D163B2: R__zip (Bits.h:441)
==28682==    by 0xBBC2B2: TBasket::WriteBuffer() (TBasket.cxx:847)
==28682==    by 0xBC42D3: TBranch::WriteBasket(TBasket*, int) (TBranch.cxx:2490)
==28682==    by 0xBC4FF0: TBranch::Fill() (TBranch.cxx:920)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682== 
==28682== Conditional jump or move depends on uninitialised value(s)
==28682==    at 0x829BEE9: deflateEnd (in /usr/lib/libz.1.2.3.dylib)
==28682==    by 0x829CDE7: deflateSetDictionary (in /usr/lib/libz.1.2.3.dylib)
==28682==    by 0x829DADB: deflate (in /usr/lib/libz.1.2.3.dylib)
==28682==    by 0x1D163B2: R__zip (Bits.h:441)
==28682==    by 0xBBC2B2: TBasket::WriteBuffer() (TBasket.cxx:847)
==28682==    by 0xBC42D3: TBranch::WriteBasket(TBasket*, int) (TBranch.cxx:2490)
==28682==    by 0xBC4FF0: TBranch::Fill() (TBranch.cxx:920)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682== 
==28682== Invalid read of size 4
==28682==    at 0x2B2B60: std::_Bit_reference::operator=(bool) (stl_bvector.h:87)
==28682==    by 0x12F2EB3D: AlgRmMu::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMu.cxx:314)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F3A6FE: CandRmMu::CandRmMu(AlgHandle&, CandHandle&, CandContext&) (CandRmMu.cxx:109)
==28682==    by 0x12F3A780: CandRmMu::MakeCandidate(AlgHandle&, CandContext&) (CandRmMu.cxx:344)
==28682==    by 0x12F34D90: AlgRmMuList::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMuList.cxx:112)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F43822: CandRmMuList::CandRmMuList(AlgHandle&, CandHandle&, CandContext&) (CandRmMuList.cxx:61)
==28682==    by 0x12F4388E: CandRmMuList::MakeCandidate(AlgHandle&, CandContext&) (CandRmMuList.cxx:130)
==28682==    by 0x12F5DD5D: RemoveMuon::Reco(MomNavigator*) (RemoveMuon.cxx:273)
==28682==    by 0x777230B: JobCMethod::Execute(JobCModule*, MomNavigator*) const (JobCMethod.cxx:73)
==28682==    by 0x7777334: JobCNode::Execute(MomNavigator*) (JobCNode.cxx:150)
==28682==  Address 0x37856488 is 0 bytes after a block of size 8 alloc'd
==28682==    at 0x320E5: operator new(unsigned long) (vg_replace_malloc.c:220)
==28682==    by 0x2B6F3D: __gnu_cxx::new_allocator<unsigned long>::allocate(unsigned long, void const*) (new_allocator.h:91)
==28682==    by 0x2B6F67: std::_Bvector_base<std::allocator<bool> >::_M_allocate(unsigned long) (stl_bvector.h:419)
==28682==    by 0x2B6F81: std::vector<bool, std::allocator<bool> >::_M_initialize(unsigned long) (stl_bvector.h:784)
==28682==    by 0x2B700C: std::vector<bool, std::allocator<bool> >::vector(unsigned long, bool const&, std::allocator<bool> const&) (stl_bvector.h:492)
==28682==    by 0x12F2E4F0: AlgRmMu::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMu.cxx:260)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F3A6FE: CandRmMu::CandRmMu(AlgHandle&, CandHandle&, CandContext&) (CandRmMu.cxx:109)
==28682==    by 0x12F3A780: CandRmMu::MakeCandidate(AlgHandle&, CandContext&) (CandRmMu.cxx:344)
==28682==    by 0x12F34D90: AlgRmMuList::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMuList.cxx:112)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F43822: CandRmMuList::CandRmMuList(AlgHandle&, CandHandle&, CandContext&) (CandRmMuList.cxx:61)
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x2B2B6A: std::_Bit_reference::operator=(bool) (stl_bvector.h:87)
==28682==    by 0x12F2EB3D: AlgRmMu::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMu.cxx:314)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F3A6FE: CandRmMu::CandRmMu(AlgHandle&, CandHandle&, CandContext&) (CandRmMu.cxx:109)
==28682==    by 0x12F3A780: CandRmMu::MakeCandidate(AlgHandle&, CandContext&) (CandRmMu.cxx:344)
==28682==    by 0x12F34D90: AlgRmMuList::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMuList.cxx:112)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F43822: CandRmMuList::CandRmMuList(AlgHandle&, CandHandle&, CandContext&) (CandRmMuList.cxx:61)
==28682==    by 0x12F4388E: CandRmMuList::MakeCandidate(AlgHandle&, CandContext&) (CandRmMuList.cxx:130)
==28682==    by 0x12F5DD5D: RemoveMuon::Reco(MomNavigator*) (RemoveMuon.cxx:273)
==28682==    by 0x777230B: JobCMethod::Execute(JobCModule*, MomNavigator*) const (JobCMethod.cxx:73)
==28682==    by 0x7777334: JobCNode::Execute(MomNavigator*) (JobCNode.cxx:150)
==28682==  Address 0x37856488 is 0 bytes after a block of size 8 alloc'd
==28682==    at 0x320E5: operator new(unsigned long) (vg_replace_malloc.c:220)
==28682==    by 0x2B6F3D: __gnu_cxx::new_allocator<unsigned long>::allocate(unsigned long, void const*) (new_allocator.h:91)
==28682==    by 0x2B6F67: std::_Bvector_base<std::allocator<bool> >::_M_allocate(unsigned long) (stl_bvector.h:419)
==28682==    by 0x2B6F81: std::vector<bool, std::allocator<bool> >::_M_initialize(unsigned long) (stl_bvector.h:784)
==28682==    by 0x2B700C: std::vector<bool, std::allocator<bool> >::vector(unsigned long, bool const&, std::allocator<bool> const&) (stl_bvector.h:492)
==28682==    by 0x12F2E4F0: AlgRmMu::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMu.cxx:260)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F3A6FE: CandRmMu::CandRmMu(AlgHandle&, CandHandle&, CandContext&) (CandRmMu.cxx:109)
==28682==    by 0x12F3A780: CandRmMu::MakeCandidate(AlgHandle&, CandContext&) (CandRmMu.cxx:344)
==28682==    by 0x12F34D90: AlgRmMuList::RunAlg(AlgConfig&, CandHandle&, CandContext&) (AlgRmMuList.cxx:112)
==28682==    by 0x601CEFE: AlgHandle::RunAlg(CandHandle&, CandContext&) (AlgHandle.cxx:68)
==28682==    by 0x12F43822: CandRmMuList::CandRmMuList(AlgHandle&, CandHandle&, CandContext&) (CandRmMuList.cxx:61)
==28682== 
RunAlg, New track, Run: 17725 Snarl: 100030
=D= AlgFitTr [17725|100030] AlgFitTrackCam.cxx:3304> GenerateNDSpectStrips
RunTheFitter, Call methods in the appropriate order

...

=D= AlgFitTr [17725|101788] AlgFitTrackCam.cxx:2207>  Checking Last Plane: Q/P=1.76149 MajorityCurvature = 1
 StoreData [66]  x_k[4] = 1.76149 Entry=4
GoForwards, Finished
 StoreData [66]  x_k[4] = 1.76149 Entry=5
GoBackwards, carry out fit in negative z direction
Fitting from StartPlane=66 to EndPlane=41 end of range plane = 66
 StoreData [61]  x_k[4] = 1.36703 Entry=5
 StoreData [51]  x_k[4] = 0.93493 Entry=5
 StoreData [46]  x_k[4] = 0.802988 Entry=5
 StoreData [41]  x_k[4] = 0.689214 Entry=5
GoBackwards, Finished
Set Track Properties:
 Q/P = 0.698174, Q/P(biased) = 0.867209, sigma(Q/P) = 0.21357 Sigma Q/P _biased 0.22687
SetRangeAnddS from swimmer values between 41/66
==28682== Invalid write of size 2
==28682==    at 0x170B5FC: tobuf(char*&, unsigned short) (Bytes.h:74)
==28682==    by 0x1733386: tobuf(char*&, short) (Bytes.h:440)
==28682==    by 0x170C6CB: TBufferFile::WriteShort(short) (TBufferFile.h:346)
==28682==    by 0x5F8FE: operator<<(TBuffer&, short) (TBuffer.h:342)
==28682==    by 0x1708051: TBufferFile::WriteVersion(TClass const*, bool) (TBufferFile.cxx:2878)
==28682==    by 0x1831BDD: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:610)
==28682==    by 0xBDB259: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1281)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==  Address 0x18a024bd is not stack'd, malloc'd or (recently) free'd
==28682== 
RunAlg, New track, Run: 17725 Snarl: 101788
=D= AlgFitTr [17725|101788] AlgFitTrackCam.cxx:3304> GenerateNDSpectStrips
RunTheFitter, Call methods in the appropriate order

...

=D= AlgFitTr [17725|103898] AlgFitTrackCam.cxx:2207>  Checking Last Plane: Q/P=-1.35551 MajorityCurvature = -1
 StoreData [15]  x_k[4] = -1.35551 Entry=4
GoForwards, Finished
 StoreData [15]  x_k[4] = -1.35551 Entry=5
GoBackwards, carry out fit in negative z direction
Fitting from StartPlane=15 to EndPlane=6 end of range plane = 15
 StoreData [14]  x_k[4] = -1.28519 Entry=5
 StoreData [13]  x_k[4] = -1.22289 Entry=5
 StoreData [12]  x_k[4] = -1.16066 Entry=5
 StoreData [11]  x_k[4] = -1.10178 Entry=5
 StoreData [10]  x_k[4] = -1.19132 Entry=5
 StoreData [9]  x_k[4] = -1.12373 Entry=5
 StoreData [7]  x_k[4] = -1.00114 Entry=5
 StoreData [6]  x_k[4] = -1.12377 Entry=5
GoBackwards, Finished
Set Track Properties:
 Q/P = -1.13838, Q/P(biased) = -1.13025, sigma(Q/P) = 0.427513 Sigma Q/P _biased 0.516891
SetRangeAnddS from swimmer values between 6/15
==28682== Invalid write of size 1
==28682==    at 0x170B5D6: tobuf(char*&, unsigned char) (Bytes.h:67)
==28682==    by 0x1733361: tobuf(char*&, char) (Bytes.h:439)
==28682==    by 0x170C71B: TBufferFile::WriteUChar(unsigned char) (TBufferFile.h:339)
==28682==    by 0xC0FA5C: operator<<(TBuffer&, unsigned char) (TBuffer.h:341)
==28682==    by 0x1C69D63: TString::Streamer(TBuffer&) (TString.cxx:1169)
==28682==    by 0x174B8A8: TGenCollectionStreamer::WriteMap(int, TBuffer&) (TGenCollectionStreamer.cxx:749)
==28682==    by 0x174EC20: TGenCollectionStreamer::Streamer(TBuffer&) (TGenCollectionStreamer.cxx:1028)
==28682==    by 0x170D4A4: TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) (TCollectionProxyFactory.cxx:183)
==28682==    by 0x170E276: TCollectionClassStreamer::Stream(TBuffer&, void*, TClass const*) (TCollectionProxyFactory.h:198)
==28682==    by 0x1CDA417: TClass::StreamerExternal(void*, TBuffer&, TClass const*) const (TClass.cxx:4961)
==28682==    by 0xBE6456: TClass::Streamer(void*, TBuffer&, TClass const*) const (TClass.h:375)
==28682==    by 0x170A02E: TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) (TBufferFile.cxx:2128)
==28682==  Address 0x4e8d01cd is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33983: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x1C69D93: TString::Streamer(TBuffer&) (TString.cxx:1172)
==28682==    by 0x174B8A8: TGenCollectionStreamer::WriteMap(int, TBuffer&) (TGenCollectionStreamer.cxx:749)
==28682==    by 0x174EC20: TGenCollectionStreamer::Streamer(TBuffer&) (TGenCollectionStreamer.cxx:1028)
==28682==    by 0x170D4A4: TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) (TCollectionProxyFactory.cxx:183)
==28682==    by 0x170E276: TCollectionClassStreamer::Stream(TBuffer&, void*, TClass const*) (TCollectionProxyFactory.h:198)
==28682==    by 0x1CDA417: TClass::StreamerExternal(void*, TBuffer&, TClass const*) const (TClass.cxx:4961)
==28682==    by 0xBE6456: TClass::Streamer(void*, TBuffer&, TClass const*) const (TClass.h:375)
==28682==    by 0x170A02E: TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) (TBufferFile.cxx:2128)
==28682==    by 0x1831C4F: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:613)
==28682==    by 0xBDB259: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1281)
==28682==  Address 0x46c8a0ee is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33989: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x1C69D93: TString::Streamer(TBuffer&) (TString.cxx:1172)
==28682==    by 0x174B8A8: TGenCollectionStreamer::WriteMap(int, TBuffer&) (TGenCollectionStreamer.cxx:749)
==28682==    by 0x174EC20: TGenCollectionStreamer::Streamer(TBuffer&) (TGenCollectionStreamer.cxx:1028)
==28682==    by 0x170D4A4: TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) (TCollectionProxyFactory.cxx:183)
==28682==    by 0x170E276: TCollectionClassStreamer::Stream(TBuffer&, void*, TClass const*) (TCollectionProxyFactory.h:198)
==28682==    by 0x1CDA417: TClass::StreamerExternal(void*, TBuffer&, TClass const*) const (TClass.cxx:4961)
==28682==    by 0xBE6456: TClass::Streamer(void*, TBuffer&, TClass const*) const (TClass.h:375)
==28682==    by 0x170A02E: TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) (TBufferFile.cxx:2128)
==28682==    by 0x1831C4F: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:613)
==28682==    by 0xBDB259: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1281)
==28682==  Address 0x46c8a0ef is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33990: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x1C69D93: TString::Streamer(TBuffer&) (TString.cxx:1172)
==28682==    by 0x174B8A8: TGenCollectionStreamer::WriteMap(int, TBuffer&) (TGenCollectionStreamer.cxx:749)
==28682==    by 0x174EC20: TGenCollectionStreamer::Streamer(TBuffer&) (TGenCollectionStreamer.cxx:1028)
==28682==    by 0x170D4A4: TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) (TCollectionProxyFactory.cxx:183)
==28682==    by 0x170E276: TCollectionClassStreamer::Stream(TBuffer&, void*, TClass const*) (TCollectionProxyFactory.h:198)
==28682==    by 0x1CDA417: TClass::StreamerExternal(void*, TBuffer&, TClass const*) const (TClass.cxx:4961)
==28682==    by 0xBE6456: TClass::Streamer(void*, TBuffer&, TClass const*) const (TClass.h:375)
==28682==    by 0x170A02E: TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) (TBufferFile.cxx:2128)
==28682==    by 0x1831C4F: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:613)
==28682==    by 0xBDB259: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1281)
==28682==  Address 0x46c8a0f0 is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33997: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x1C69D93: TString::Streamer(TBuffer&) (TString.cxx:1172)
==28682==    by 0x174B8A8: TGenCollectionStreamer::WriteMap(int, TBuffer&) (TGenCollectionStreamer.cxx:749)
==28682==    by 0x174EC20: TGenCollectionStreamer::Streamer(TBuffer&) (TGenCollectionStreamer.cxx:1028)
==28682==    by 0x170D4A4: TCollectionStreamer::Streamer(TBuffer&, void*, int, TClass*) (TCollectionProxyFactory.cxx:183)
==28682==    by 0x170E276: TCollectionClassStreamer::Stream(TBuffer&, void*, TClass const*) (TCollectionProxyFactory.h:198)
==28682==    by 0x1CDA417: TClass::StreamerExternal(void*, TBuffer&, TClass const*) const (TClass.cxx:4961)
==28682==    by 0xBE6456: TClass::Streamer(void*, TBuffer&, TClass const*) const (TClass.h:375)
==28682==    by 0x170A02E: TBufferFile::WriteFastArray(void*, TClass const*, int, TMemberStreamer*) (TBufferFile.cxx:2128)
==28682==    by 0x1831C4F: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:613)
==28682==    by 0xBDB259: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1281)
==28682==  Address 0x46c8a0f1 is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170B629: tobuf(char*&, unsigned int) (Bytes.h:92)
==28682==    by 0x170B9F9: tobuf(char*&, int) (Bytes.h:441)
==28682==    by 0x170C5E1: TBufferFile::WriteUInt(unsigned int) (TBufferFile.h:367)
==28682==    by 0x5F94A: operator<<(TBuffer&, unsigned int) (TBuffer.h:345)
==28682==    by 0x182DC2D: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:237)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==  Address 0x4e7280ad is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170B629: tobuf(char*&, unsigned int) (Bytes.h:92)
==28682==    by 0x170B9F9: tobuf(char*&, int) (Bytes.h:441)
==28682==    by 0x170C629: TBufferFile::WriteInt(int) (TBufferFile.h:360)
==28682==    by 0x5F924: operator<<(TBuffer&, int) (TBuffer.h:344)
==28682==    by 0x182D9B2: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:230)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==  Address 0x46d3570d is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x170B5D6: tobuf(char*&, unsigned char) (Bytes.h:67)
==28682==    by 0x1733361: tobuf(char*&, char) (Bytes.h:439)
==28682==    by 0x170C768: TBufferFile::WriteChar(char) (TBufferFile.h:332)
==28682==    by 0xBBF0F7: operator<<(TBuffer&, char) (TBuffer.h:340)
==28682==    by 0x182D906: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:228)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==  Address 0x4f7ecf7e is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 2
==28682==    at 0x170B5FC: tobuf(char*&, unsigned short) (Bytes.h:74)
==28682==    by 0x1733386: tobuf(char*&, short) (Bytes.h:440)
==28682==    by 0x170C67A: TBufferFile::WriteUShort(unsigned short) (TBufferFile.h:353)
==28682==    by 0xC16A27: operator<<(TBuffer&, unsigned short) (TBuffer.h:343)
==28682==    by 0x182DBD5: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:236)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==  Address 0x4608704c is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170B811: tobuf(char*&, float) (Bytes.h:233)
==28682==    by 0x170C451: TBufferFile::WriteFloat(float) (TBufferFile.h:402)
==28682==    by 0x5F970: operator<<(TBuffer&, float) (TBuffer.h:350)
==28682==    by 0x182DAC1: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:233)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==  Address 0x46c696c8 is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170BF9C: bswapcpy32(void*, void const*, unsigned long) (Bswapcpy.h:78)
==28682==    by 0x1704147: TBufferFile::WriteFastArray(int const*, int) (TBufferFile.cxx:1900)
==28682==    by 0x182FC1F: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:348)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==    by 0x7419748: PerOutputStreamManager::Put(MomNavigator const*) (PerOutputStreamManager.cxx:198)
==28682==  Address 0x1a7e368b is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170BF9C: bswapcpy32(void*, void const*, unsigned long) (Bswapcpy.h:78)
==28682==    by 0x1703F6D: TBufferFile::WriteFastArray(float const*, int) (TBufferFile.cxx:1971)
==28682==    by 0x182FFB5: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:351)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==    by 0x7419748: PerOutputStreamManager::Put(MomNavigator const*) (PerOutputStreamManager.cxx:198)
==28682==  Address 0x4e8b5ed7 is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170B880: tobuf(char*&, double) (Bytes.h:260)
==28682==    by 0x1703F01: TBufferFile::WriteFastArray(double const*, int) (TBufferFile.cxx:1995)
==28682==    by 0x18300E7: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:352)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==    by 0x7419748: PerOutputStreamManager::Put(MomNavigator const*) (PerOutputStreamManager.cxx:198)
==28682==  Address 0x1fbbc223 is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170B882: tobuf(char*&, double) (Bytes.h:260)
==28682==    by 0x1703F01: TBufferFile::WriteFastArray(double const*, int) (TBufferFile.cxx:1995)
==28682==    by 0x18300E7: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:352)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==    by 0x7419748: PerOutputStreamManager::Put(MomNavigator const*) (PerOutputStreamManager.cxx:198)
==28682==  Address 0x1fbbc227 is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33913: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x170C984: TBufferFile::WriteFastArray(unsigned char const*, int) (TBufferFile.h:568)
==28682==    by 0x1830219: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:353)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==  Address 0x4535145c is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33919: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x170C984: TBufferFile::WriteFastArray(unsigned char const*, int) (TBufferFile.h:568)
==28682==    by 0x1830219: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:353)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==  Address 0x4535145b is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33920: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x170C984: TBufferFile::WriteFastArray(unsigned char const*, int) (TBufferFile.h:568)
==28682==    by 0x1830219: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:353)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==  Address 0x4535145a is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 1
==28682==    at 0x33927: memcpy (mc_replace_strmem.c:482)
==28682==    by 0x1706DE0: TBufferFile::WriteFastArray(char const*, int) (TBufferFile.cxx:1839)
==28682==    by 0x170C984: TBufferFile::WriteFastArray(unsigned char const*, int) (TBufferFile.h:568)
==28682==    by 0x1830219: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:353)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==  Address 0x45351459 is not stack'd, malloc'd or (recently) free'd
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170B880: tobuf(char*&, double) (Bytes.h:260)
==28682==    by 0x170C409: TBufferFile::WriteDouble(double) (TBufferFile.h:409)
==28682==    by 0x171EBC6: operator<<(TBuffer&, double) (TBuffer.h:351)
==28682==    by 0x182DB1D: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:234)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==  Address 0x33aeada5 is 9 bytes after a block of size 28 alloc'd
==28682==    at 0x320E5: operator new(unsigned long) (vg_replace_malloc.c:220)
==28682==    by 0x4F2F2E8: __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::allocate(unsigned long, void const*) (new_allocator.h:91)
==28682==    by 0x4F2F30C: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_get_node() (stl_tree.h:369)
==28682==    by 0x4F2F42D: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_create_node(std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_tree.h:378)
==28682==    by 0x4F2F54C: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_insert(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_tree.h:840)
==28682==    by 0x4F2F9CC: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_insert_unique(std::_Rb_tree_iterator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_tree.h:1026)
==28682==    by 0x4F2FB86: std::map<PlexPixelSpotId, CalADCtoPEs const*, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::insert(std::_Rb_tree_iterator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_map.h:427)
==28682==    by 0x4F2FC5A: std::map<PlexPixelSpotId, CalADCtoPEs const*, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::operator[](PlexPixelSpotId const&) (stl_map.h:350)
==28682==    by 0x4F2B1CD: PEGainAggCalScheme::GetRow(PlexStripEndId const&) const (PEGainAggCalScheme.cxx:344)
==28682==    by 0x4F2B752: PEGainAggCalScheme::GetPhotoElectrons(ValueErr<float>, PlexStripEndId const&) const (PEGainAggCalScheme.cxx:156)
==28682==    by 0x4F20482: Calibrator::GetPhotoElectrons(ValueErr<float>, PlexStripEndId const&) const (Calibrator.cxx:266)
==28682==    by 0x4F21637: Calibrator::CalibStripEnd(PlexStripEndId const&, PlexPixelSpotId const&, int, double) const (Calibrator.cxx:66)
==28682== 
==28682== Invalid write of size 4
==28682==    at 0x170B882: tobuf(char*&, double) (Bytes.h:260)
==28682==    by 0x170C409: TBufferFile::WriteDouble(double) (TBufferFile.h:409)
==28682==    by 0x171EBC6: operator<<(TBuffer&, double) (TBuffer.h:351)
==28682==    by 0x182DB1D: int TStreamerInfo::WriteBufferAux<char**>(TBuffer&, char** const&, int, int, int, int) (TStreamerInfoWriteBuffer.cxx:234)
==28682==    by 0x1826284: TStreamerInfo::WriteBufferClones(TBuffer&, TClonesArray*, int, int, int) (TStreamerInfoWriteBuffer.cxx:900)
==28682==    by 0xBDBADC: TBranchElement::FillLeaves(TBuffer&) (TBranchElement.cxx:1400)
==28682==    by 0xBC4EC6: TBranch::Fill() (TBranch.cxx:893)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==  Address 0x33aeada9 is 13 bytes after a block of size 28 alloc'd
==28682==    at 0x320E5: operator new(unsigned long) (vg_replace_malloc.c:220)
==28682==    by 0x4F2F2E8: __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::allocate(unsigned long, void const*) (new_allocator.h:91)
==28682==    by 0x4F2F30C: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_get_node() (stl_tree.h:369)
==28682==    by 0x4F2F42D: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_create_node(std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_tree.h:378)
==28682==    by 0x4F2F54C: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_insert(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_tree.h:840)
==28682==    by 0x4F2F9CC: std::_Rb_tree<PlexPixelSpotId, std::pair<PlexPixelSpotId const, CalADCtoPEs const*>, std::_Select1st<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::_M_insert_unique(std::_Rb_tree_iterator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_tree.h:1026)
==28682==    by 0x4F2FB86: std::map<PlexPixelSpotId, CalADCtoPEs const*, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::insert(std::_Rb_tree_iterator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> >, std::pair<PlexPixelSpotId const, CalADCtoPEs const*> const&) (stl_map.h:427)
==28682==    by 0x4F2FC5A: std::map<PlexPixelSpotId, CalADCtoPEs const*, std::less<PlexPixelSpotId>, std::allocator<std::pair<PlexPixelSpotId const, CalADCtoPEs const*> > >::operator[](PlexPixelSpotId const&) (stl_map.h:350)
==28682==    by 0x4F2B1CD: PEGainAggCalScheme::GetRow(PlexStripEndId const&) const (PEGainAggCalScheme.cxx:344)
==28682==    by 0x4F2B752: PEGainAggCalScheme::GetPhotoElectrons(ValueErr<float>, PlexStripEndId const&) const (PEGainAggCalScheme.cxx:156)
==28682==    by 0x4F20482: Calibrator::GetPhotoElectrons(ValueErr<float>, PlexStripEndId const&) const (Calibrator.cxx:266)
==28682==    by 0x4F21637: Calibrator::CalibStripEnd(PlexStripEndId const&, PlexPixelSpotId const&, int, double) const (Calibrator.cxx:66)
==28682== 
--28682-- VALGRIND INTERNAL ERROR: Valgrind received a signal 11 (SIGSEGV) - exiting
--28682-- si_code=1;  Faulting address: 0x8E9BE112;  sp: 0xf129adb0

valgrind: the 'impossible' happened:
   Killed by fatal signal
==28682==    at 0xF00B678C: ???
==28682==    by 0xF0102908: ???
==28682==    by 0xF0082C8B: ???
==28682==    by 0xF0083727: ???
==28682==    by 0xF0105034: ???
==28682==    by 0xF012B3DF: ???

sched status:
  running_tid=1

Thread 1: status = VgTs_Runnable
==28682==    at 0x30F18: free (vg_replace_malloc.c:325)
==28682==    by 0x829BDAF: deflateEnd (in /usr/lib/libz.1.2.3.dylib)
==28682==    by 0x1D163D6: R__zip (Bits.h:451)
==28682==    by 0xBBC2B2: TBasket::WriteBuffer() (TBasket.cxx:847)
==28682==    by 0xBC42D3: TBranch::WriteBasket(TBasket*, int) (TBranch.cxx:2490)
==28682==    by 0xBC4FF0: TBranch::Fill() (TBranch.cxx:920)
==28682==    by 0xBDBDF2: TBranchElement::Fill() (TBranchElement.cxx:1176)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xBDBF65: TBranchElement::Fill() (TBranchElement.cxx:1201)
==28682==    by 0xC2A5D2: TTree::Fill() (TTree.cxx:3919)
==28682==    by 0x74181EF: PerOutputStream::Store() (PerOutputStream.cxx:624)
==28682==    by 0x7419748: PerOutputStreamManager::Put(MomNavigator const*) (PerOutputStreamManager.cxx:198)
==28682==    by 0x6EB5288: IoOutputModule::Put(MomNavigator const*) (IoOutputModule.cxx:194)
==28682==    by 0x77722C9: JobCMethod::Execute(JobCModule*, MomNavigator*) const (JobCMethod.cxx:71)
==28682==    by 0x7777334: JobCNode::Execute(MomNavigator*) (JobCNode.cxx:150)
==28682==    by 0x777A70E: JobCPath::Execute(JobCRecord*) (JobCPath.cxx:807)
==28682==    by 0x777A951: JobCPath::EvalInput(JobCRecord*) (JobCPath.cxx:739)
==28682==    by 0x776DB86: JobCGraphVtx::EvaluateNode(JobCRecord*) (JobCGraphVtx.cxx:63)
==28682==    by 0x776DBF3: JobCGraphVtx::EvaluateNode(JobCRecord*) (JobCGraphVtx.cxx:68)
==28682==    by 0x776DC3A: JobCGraphVtx::EvaluateGraph(JobCRecord*) (JobCGraphVtx.cxx:41)
==28682==    by 0x777A3BC: JobCPath::Run(int, JobCPath::ERunMode) (JobCPath.cxx:943)
==28682==    by 0x77B004E: G__JobControlCint_430_0_6(G__value*, char const*, G__param*, int) (JobControlCint.cc:1997)
==28682==    by 0x2737B68: Cint::G__ExceptionWrapper(int (*)(G__value*, char const*, G__param*, int), G__value*, char*, G__param*, int) (Api.cxx:393)
==28682==    by 0x2836AD8: G__execute_call (newlink.cxx:2406)
==28682==    by 0x28374DF: G__call_cppfunc (newlink.cxx:2610)
==28682==    by 0x27F3B99: G__interpret_func (ifunc.cxx:5795)
==28682==    by 0x27E0156: G__getfunction (func.cxx:2665)
==28682==    by 0x28F57E9: G__getstructmem(int, G__FastAllocString&, char*, int, char*, int*, G__var_array*, int) (var.cxx:6811)
==28682==    by 0x28E4F63: G__getvariable (var.cxx:5380)
==28682==    by 0x27A4EC1: G__getitem (expr.cxx:1906)
==28682==    by 0x27BB15C: G__getexpr (expr.cxx:1488)
==28682==    by 0x28523C0: G__exec_function(G__FastAllocString&, int*, int*, int*, G__value*) (parse.cxx:645)
==28682==    by 0x285B60E: G__exec_statement (parse.cxx:7372)
==28682==    by 0x28649B8: G__exec_if (parse.cxx:1740)
==28682==    by 0x285AFB1: G__exec_statement (parse.cxx:7286)
==28682==    by 0x27F7118: G__interpret_func (ifunc.cxx:6714)
==28682==    by 0x27E1033: G__getfunction (func.cxx:2906)
==28682==    by 0x27A5039: G__getitem (expr.cxx:1918)
==28682==    by 0x27BB15C: G__getexpr (expr.cxx:1488)
==28682==    by 0x28523C0: G__exec_function(G__FastAllocString&, int*, int*, int*, G__value*) (parse.cxx:645)
==28682==    by 0x285B60E: G__exec_statement (parse.cxx:7372)
==28682==    by 0x27F7118: G__interpret_func (ifunc.cxx:6714)
==28682==    by 0x27E1033: G__getfunction (func.cxx:2906)
==28682==    by 0x27A5039: G__getitem (expr.cxx:1918)
==28682==    by 0x27BB15C: G__getexpr (expr.cxx:1488)
==28682==    by 0x27BCE82: G__calc_internal (expr.cxx:1071)
==28682==    by 0x286CEE5: G__process_cmd (pause.cxx:2315)
==28682==    by 0x1BBCBD7: TCint::ProcessLine(char const*, TInterpreter::EErrorCode*) (TCint.cxx:522)
==28682==    by 0x1CC6631: TCint::ProcessLineSynch(char const*, TInterpreter::EErrorCode*) (TCint.cxx:601)
==28682==    by 0x1C1622A: TApplication::ExecuteFile(char const*, int*, bool) (TApplication.cxx:1035)
==28682==    by 0x1C163E1: TApplication::ProcessFile(char const*, int*, bool) (TApplication.cxx:911)
==28682==    by 0x1C19512: TApplication::ProcessLine(char const*, bool, int*) (TApplication.cxx:884)
==28682==    by 0xE73456: TRint::Run(bool) (TRint.cxx:396)
==28682==    by 0x7788E5B: JobCRootEnv::RunTheApp() (JobCRootEnv.cxx:290)
==28682==    by 0x77692E5: JobCEnv::RunRootApp() (JobCEnv.cxx:396)
==28682==    by 0x432BE25: main (JobCmain.cxx:22)

Note: see also the FAQ in the source distribution.
It contains workarounds to several common problems.
In particular, if Valgrind aborted or crashed after
identifying problems in your program, there's a good chance
that fixing those problems will prevent Valgrind aborting or
crashing, especially if it happened in m_mallocfree.c.

If that doesn't help, please report this bug to: www.valgrind.org

In the bug report, send all the above text, the valgrind
version, and what OS and version you are using.  Thanks.

Leopard> 



Also available in: Atom PDF