Project

General

Profile

Support #20458

Summary of EventBuilder crashes at protoDUNE July 23-26

Added by Kurt Biery 12 months ago. Updated 12 months ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
Start date:
07/26/2018
Due date:
% Done:

0%

Estimated time:
Experiment:
-
Co-Assignees:
Duration:

History

#1 Updated by Kurt Biery 12 months ago

EventBuilder2 on srv-001:

%MSG-i EventBuilder2_TCPSocketTransfer: Initializing 24-Jul-2018 06:10:06 CEST Booted TCPSocket_transfer.cc:790
transfer_between_0_and_10_RECV: Starting Listener Thread for port 11010 (rank=10,partition=1)
%MSG
%MSG-e EventBuilder2_TCP_listen_fd: Initializing 24-Jul-2018 06:10:06 CEST Booted TCP_listen_fd.cc:55
Could not bind socket for port 11010! Exiting with code 3!
%MSG
bind error: File exists

  • Break *** segmentation violation
    /nfs/home/np04daq/boot.sh: line 10: 143150 Segmentation fault (core dumped) $2 -c "id: $3 commanderPluginType: xmlrpc rank: $4 application_name: $5 partition_number: $6"

#2 Updated by Kurt Biery 12 months ago

EventBuilder6 on srv-002 at 23:24:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `eventbuilder -c id: 6239 commanderPluginType: xmlrpc rank: 77 application_name:'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  artdaq::SharedMemoryManager::GetBuffersOwnedByManager (this=this@entry=0x7f3c0800b420, locked=locked@entry=false)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:499
499            for (auto ii = 0; ii < shm_ptr_->buffer_count; ++ii)
[Current thread is 1 (Thread 0x7f3c1f38b4c0 (LWP 80625))]
(gdb) where
#0  artdaq::SharedMemoryManager::GetBuffersOwnedByManager (this=this@entry=0x7f3c0800b420, locked=locked@entry=false)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:499
#1  0x00007f3c1d31e9f5 in artdaq::SharedMemoryManager::Detach (this=this@entry=0x7f3c0800b420, throwException=throwException@entry=false, category=..., 
    message=..., force=force@entry=true) at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:945
#2  0x00007f3c1d339f3e in signal_handler (signum=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:29
#3  <signal handler called>
#4  0x00007f3c1a3aaa3d in poll () from /lib64/libc.so.6
#5  0x00007f3c11cc6c3a in waitForConnection (listenSocketP=0x14590c0, listenSocketP=0x14590c0, errorP=0x7ffc1d64dba0, interruptedP=<synthetic pointer>)
    at socket_unix.c:694
#6  chanSwitchAccept (chanSwitchP=<optimized out>, channelPP=0x7ffc1d64dba8, channelInfoPP=0x7ffc1d64dbb0, errorP=0x7ffc1d64dba0) at socket_unix.c:804
#7  0x00007f3c11cbda7f in ChanSwitchAccept (chanSwitchP=0x14590e0, channelPP=0x7ffc1d64dba8, channelInfoPP=0x7ffc1d64dbb0, errorP=<optimized out>)
    at chanswitch.c:159
#8  0x00007f3c11cc54ef in acceptAndProcessNextConnection (errorP=0x7ffc1d64db98, outstandingConnListP=0x1458fb0, serverP=0x1459090) at server.c:1191
#9  serverRun2 (errorP=0x7ffc1d64db98, serverP=0x1459090) at server.c:1242
#10 ServerRun (serverP=serverP@entry=0x1459090) at server.c:1280
#11 0x00007f3c127063f8 in xmlrpc_c::setupSignalsAndRunAbyss (abyssServerP=0x1459090) at server_abyss.cpp:760
#12 0x00007f3c12707219 in xmlrpc_c::serverAbyss_impl::run (this=<optimized out>) at server_abyss.cpp:771
#13 0x00007f3c127076bd in xmlrpc_c::serverAbyss::run (this=<optimized out>) at server_abyss.cpp:873
#14 0x00007f3c1292e0b3 in artdaq::xmlrpc_commander::run_server (this=0x1455d60)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/artdaq/ExternalComms/xmlrpc_commander.cc:1114
#15 0x000000000042a53c in artdaq::artdaqapp::runArtdaqApp (task=task@entry=artdaq::detail::EventBuilderTask, config_ps=...)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/proto/artdaqapp.hh:113
#16 0x00000000004167cd in main (argc=3, argv=0x7ffc1d65b4d8) at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/proto/eventbuilder.cc:9
(gdb) 
#0  artdaq::SharedMemoryManager::GetBuffersOwnedByManager (this=this@entry=0x7f3c0800b420, locked=locked@entry=false)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:499
#1  0x00007f3c1d31e9f5 in artdaq::SharedMemoryManager::Detach (this=this@entry=0x7f3c0800b420, throwException=throwException@entry=false, category=..., 
    message=..., force=force@entry=true) at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:945
#2  0x00007f3c1d339f3e in signal_handler (signum=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:29
#3  <signal handler called>
#4  0x00007f3c1a3aaa3d in poll () from /lib64/libc.so.6
#5  0x00007f3c11cc6c3a in waitForConnection (listenSocketP=0x14590c0, listenSocketP=0x14590c0, errorP=0x7ffc1d64dba0, interruptedP=<synthetic pointer>)
    at socket_unix.c:694
#6  chanSwitchAccept (chanSwitchP=<optimized out>, channelPP=0x7ffc1d64dba8, channelInfoPP=0x7ffc1d64dbb0, errorP=0x7ffc1d64dba0) at socket_unix.c:804
#7  0x00007f3c11cbda7f in ChanSwitchAccept (chanSwitchP=0x14590e0, channelPP=0x7ffc1d64dba8, channelInfoPP=0x7ffc1d64dbb0, errorP=<optimized out>)
    at chanswitch.c:159
#8  0x00007f3c11cc54ef in acceptAndProcessNextConnection (errorP=0x7ffc1d64db98, outstandingConnListP=0x1458fb0, serverP=0x1459090) at server.c:1191
#9  serverRun2 (errorP=0x7ffc1d64db98, serverP=0x1459090) at server.c:1242
#10 ServerRun (serverP=serverP@entry=0x1459090) at server.c:1280
#11 0x00007f3c127063f8 in xmlrpc_c::setupSignalsAndRunAbyss (abyssServerP=0x1459090) at server_abyss.cpp:760
#12 0x00007f3c12707219 in xmlrpc_c::serverAbyss_impl::run (this=<optimized out>) at server_abyss.cpp:771
#13 0x00007f3c127076bd in xmlrpc_c::serverAbyss::run (this=<optimized out>) at server_abyss.cpp:873
#14 0x00007f3c1292e0b3 in artdaq::xmlrpc_commander::run_server (this=0x1455d60)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/artdaq/ExternalComms/xmlrpc_commander.cc:1114
#15 0x000000000042a53c in artdaq::artdaqapp::runArtdaqApp (task=task@entry=artdaq::detail::EventBuilderTask, config_ps=...)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/proto/artdaqapp.hh:113
#16 0x00000000004167cd in main (argc=3, argv=0x7ffc1d65b4d8) at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/proto/eventbuilder.cc:9
(gdb) 
#0  artdaq::SharedMemoryManager::GetBuffersOwnedByManager (this=this@entry=0x7f3c0800b420, locked=locked@entry=false)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:499
#1  0x00007f3c1d31e9f5 in artdaq::SharedMemoryManager::Detach (this=this@entry=0x7f3c0800b420, throwException=throwException@entry=false, category=..., 
    message=..., force=force@entry=true) at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:945
#2  0x00007f3c1d339f3e in signal_handler (signum=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq_core/artdaq-core/Core/SharedMemoryManager.cc:29
#3  <signal handler called>
#4  0x00007f3c1a3aaa3d in poll () from /lib64/libc.so.6
#5  0x00007f3c11cc6c3a in waitForConnection (listenSocketP=0x14590c0, listenSocketP=0x14590c0, errorP=0x7ffc1d64dba0, interruptedP=<synthetic pointer>)
    at socket_unix.c:694
#6  chanSwitchAccept (chanSwitchP=<optimized out>, channelPP=0x7ffc1d64dba8, channelInfoPP=0x7ffc1d64dbb0, errorP=0x7ffc1d64dba0) at socket_unix.c:804
#7  0x00007f3c11cbda7f in ChanSwitchAccept (chanSwitchP=0x14590e0, channelPP=0x7ffc1d64dba8, channelInfoPP=0x7ffc1d64dbb0, errorP=<optimized out>)
    at chanswitch.c:159
#8  0x00007f3c11cc54ef in acceptAndProcessNextConnection (errorP=0x7ffc1d64db98, outstandingConnListP=0x1458fb0, serverP=0x1459090) at server.c:1191
#9  serverRun2 (errorP=0x7ffc1d64db98, serverP=0x1459090) at server.c:1242
#10 ServerRun (serverP=serverP@entry=0x1459090) at server.c:1280
#11 0x00007f3c127063f8 in xmlrpc_c::setupSignalsAndRunAbyss (abyssServerP=0x1459090) at server_abyss.cpp:760
#12 0x00007f3c12707219 in xmlrpc_c::serverAbyss_impl::run (this=<optimized out>) at server_abyss.cpp:771
#13 0x00007f3c127076bd in xmlrpc_c::serverAbyss::run (this=<optimized out>) at server_abyss.cpp:873
#14 0x00007f3c1292e0b3 in artdaq::xmlrpc_commander::run_server (this=0x1455d60)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/artdaq/ExternalComms/xmlrpc_commander.cc:1114
#15 0x000000000042a53c in artdaq::artdaqapp::runArtdaqApp (task=task@entry=artdaq::detail::EventBuilderTask, config_ps=...)
    at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/proto/artdaqapp.hh:113
#16 0x00000000004167cd in main (argc=3, argv=0x7ffc1d65b4d8) at /nfs/sw/work_dirs/dune-artdaq-20180722/srcs/artdaq/proto/eventbuilder.cc:9

#3 Updated by Kurt Biery 12 months ago

EventBuilder10 on srv-004 at 11:02 on 23-Jul:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `eventbuilder -c id: 6244 commanderPluginType: xmlrpc rank: 16 application_name:'.
Program terminated with signal SIGABRT, Aborted.
#0  0x00007feacafe81f7 in raise () from /lib64/libc.so.6
[Current thread is 1 (Thread 0x7feabd911400 (LWP 123631))]
(gdb) where
#0  0x00007feacafe81f7 in raise () from /lib64/libc.so.6
#1  0x00007feacafe9a28 in abort () from /lib64/libc.so.6
#2  0x00007feacb027f47 in __libc_message () from /lib64/libc.so.6
#3  0x00007feacb02f619 in _int_free () from /lib64/libc.so.6
#4  0x00007fe942f457f8 in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::deallocate (this=<optimized out>, __p=0x7fe9380018c0) at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/ext/new_allocator.h:110
#5  std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > > >::deallocate (
    __a=..., __n=1, __p=0x7fe9380018c0) at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/alloc_traits.h:462
#6  std::_Rb_tree<int, std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::_M_put_node (
    __p=0x7fe9380018c0, this=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:509
#7  std::_Rb_tree<int, std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::_M_drop_node (
    __p=0x7fe9380018c0, this=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:576
#8  std::_Rb_tree<int, std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::_M_erase_aux (
    __position=..., this=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:2275
#9  std::_Rb_tree<int, std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::erase[abi:cxx11](std::_Rb_tree_const_iterator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >) (__position=..., 
    this=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:1036
#10 std::_Rb_tree<int, std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<int const, std::set<int, std::less<---Type <return> to continue, or q <return> to quit---
int>, std::allocator<int> > > >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::_M_erase_aux (
    __last=..., __first=..., this=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:2289
#11 std::_Rb_tree<int, std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::erase[abi:cxx11](std::_Rb_tree_const_iterator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >, std::_Rb_tree_const_iterator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >) (__last=..., __first=..., this=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:1069
#12 std::_Rb_tree<int, std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > >, std::_Select1st<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::erase (
    this=this@entry=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>, __x=@0x7feabd8ee1d0: 12)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:2300
#13 0x00007fe942f36f46 in std::map<int, std::set<int, std::less<int>, std::allocator<int> >, std::less<int>, std::allocator<std::pair<int const, std::set<int, std::less<int>, std::allocator<int> > > > >::erase (__x=@0x7feabd8ee1d0: 12, this=0x7fe943152920 <artdaq::TCPSocketTransfer::connected_fds_>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_map.h:981
#14 artdaq::TCPSocketTransfer::~TCPSocketTransfer (this=0x7feab8039fb0, __in_chrg=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/TransferPlugins/TCPSocket_transfer.cc:104

#15 0x00007fe942f37061 in artdaq::TCPSocketTransfer::~TCPSocketTransfer (this=0x7feab8039fb0, __in_chrg=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/TransferPlugins/TCPSocket_transfer.cc:116
#16 0x00007feabc4288e9 in std::default_delete<artdaq::TransferInterface>::operator() (this=<optimized out>, __ptr=<optimized out>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/unique_ptr.h:76
#17 std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> >::~unique_ptr (this=0x7feab803f020, __in_chrg=<optimized out>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/unique_ptr.h:239

#18 artdaq::AutodetectTransfer::~AutodetectTransfer (this=0x7feab803efd0, __in_chrg=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/TransferPlugins/Autodetect_transfer.cc:29
#19 artdaq::AutodetectTransfer::~AutodetectTransfer (this=0x7feab803efd0, __in_chrg=<optimized out>)
---Type <return> to continue, or q <return> to quit---
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/TransferPlugins/Autodetect_transfer.cc:29
#20 0x00007feace824394 in std::default_delete<artdaq::TransferInterface>::operator() (this=<optimized out>, __ptr=<optimized out>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/unique_ptr.h:76
#21 std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> >::~unique_ptr (this=<optimized out>, __in_chrg=<optimized out>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/unique_ptr.h:239
#22 std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > >::~pair (this=<optimized out>, 
    __in_chrg=<optimized out>) at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_pair.h:194
#23 __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > >::destroy<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > (this=<optimized out>, 
    __p=<optimized out>) at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/ext/new_allocator.h:124
#24 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > > >::destroy<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > (__a=..., 
    __p=<optimized out>) at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/alloc_traits.h:487
#25 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > >, std::_Select1st<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > >::_M_destroy_node (this=<optimized out>, 
    __p=0x7feab8034ec0) at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:567
#26 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > >, std::_Select1st<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > >::_M_drop_node (this=0x7feab8018b08, __p=0x7feab8034ec0)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:575
#27 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > >, std::_Select1st<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > >::_M_erase (__x=0x7feab8034ec0, this=0x7feab8018b08)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:1640
#28 std::_Rb_tree<int, std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > >, std::_Select1st<std::---Type <return> to continue, or q <return> to quit---
pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > >::_M_erase (this=0x7feab8018b08, __x=0x7feab803e150)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:1638
#29 0x00007feace814042 in std::_Rb_tree<int, std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > >, std::_Select1st<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > >::~_Rb_tree (this=0x7feab8018b08, 
    __in_chrg=<optimized out>) at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_tree.h:873
#30 std::map<int, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> >, std::less<int>, std::allocator<std::pair<int const, std::unique_ptr<artdaq::TransferInterface, std::default_delete<artdaq::TransferInterface> > > > >::~map (this=0x7feab8018b08, __in_chrg=<optimized out>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/stl_map.h:96
#31 artdaq::DataReceiverManager::~DataReceiverManager (this=0x7feab8018ac0, __in_chrg=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/DAQrate/DataReceiverManager.cc:114

#32 0x00007feace8142a1 in artdaq::DataReceiverManager::~DataReceiverManager (this=0x7feab8018ac0, __in_chrg=<optimized out>)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/DAQrate/DataReceiverManager.cc:120
#33 0x00007feacec768c7 in std::default_delete<artdaq::DataReceiverManager>::operator() (this=<optimized out>, __ptr=<optimized out>)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/unique_ptr.h:76
#34 std::unique_ptr<artdaq::DataReceiverManager, std::default_delete<artdaq::DataReceiverManager> >::reset (__p=<optimized out>, this=0x7feab801e628)
    at /nfs/sw/artdaq/products/gcc/v6_4_0/Linux64bit+3.10-2.17/include/c++/6.4.0/bits/unique_ptr.h:347
#35 artdaq::DataReceiverCore::shutdown (this=0x7feab801e620)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/Application/DataReceiverCore.cc:235
#36 0x00007feacec62d58 in artdaq::EventBuilderApp::do_shutdown (this=0x247f4e0)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/Application/EventBuilderApp.cc:96
#37 0x00007feacebf7ba7 in artdaq::Main_Initialized::shutdown (this=0x7feaceede3f0 <artdaq::Main::Initialized>, context=..., timeout=45)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/build_slf7.x86_64/artdaq/artdaq/Application/Commandable_sm.cpp:306
#38 0x00007feacebf2c5b in artdaq::CommandableContext::shutdown (timeout=45, this=0x247f4e8)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/build_slf7.x86_64/artdaq/artdaq/Application/Commandable_sm.h:296
---Type <return> to continue, or q <return> to quit---
#39 artdaq::InitializedMap_Ready::shutdown (this=<optimized out>, context=..., timeout=45)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/build_slf7.x86_64/artdaq/artdaq/Application/Commandable_sm.cpp:494
#40 0x00007feacec03c9b in artdaq::CommandableContext::shutdown (timeout=45, this=0x247f4e8)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/build_slf7.x86_64/artdaq/artdaq/Application/Commandable_sm.h:296
#41 artdaq::Commandable::shutdown (this=0x247f4e0, timeout=45)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/Application/Commandable.cc:176
#42 0x00007feac36307e6 in artdaq::shutdown_::execute_ (this=0x249dba0, paramList=...)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/ExternalComms/xmlrpc_commander.cc:611
#43 0x00007feac361d1f8 in artdaq::cmd_::execute (this=0x249dba0, paramList=..., retvalP=0x7feabd9027f8)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/ExternalComms/xmlrpc_commander.cc:411
#44 0x00007feac361b7be in c_executeMethod (envP=0x7feabd902910, paramArrayP=0x7fe8e80028e0, methodPtr=0x249dba0, callInfoPtr=0x7feabd902a30)
    at /nfs/sw/work_dirs/dune-artdaq_artdaq_v3_02_00_testing/srcs/artdaq/artdaq/ExternalComms/xmlrpc_commander.cc:81
#45 0x00007feac2bc6e3c in callNamedMethod (resultPP=0x7feabd902940, callInfoP=0x7feabd902a30, paramArrayP=0x7fe8e80028e0, methodP=<optimized out>, 
    envP=0x7feabd902910) at registry.c:307
#46 xmlrpc_dispatchCall (envP=envP@entry=0x7feabd902910, registryP=registryP@entry=0x249a7d0, methodName=0x7fe8e8001070 "daq.shutdown", 
    paramArrayP=0x7fe8e80028e0, callInfoP=callInfoP@entry=0x7feabd902a30, resultPP=resultPP@entry=0x7feabd902940) at registry.c:337
#47 0x00007feac2bc6f83 in xmlrpc_registry_process_call2 (envP=envP@entry=0x7feabd9029a0, registryP=0x249a7d0, 
    callXml=0x7fe8e8001460 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>\r\n<methodCall><methodName>daq.shutdown</methodName><params></params></methodCall>", callXmlLen=124, callInfo=callInfo@entry=0x7feabd902a30, responseXmlPP=responseXmlPP@entry=0x7feabd902990) at registry.c:426
#48 0x00007feac2fecf19 in xmlrpc_c::registry::processCall (this=0x7ffc3faa79d0, callXml=..., callInfoP=callInfoP@entry=0x7feabd902a30, 
    responseXmlP=responseXmlP@entry=0x7feabd902ac0) at registry.cpp:524
#49 0x00007feac33fd25b in xmlrpc_c::serverAbyss_impl::processCall (this=this@entry=0x249e100, call=..., abyssSessionP=abyssSessionP@entry=0x7feabd902c30, 
    responseP=responseP@entry=0x7feabd902ac0) at server_abyss.cpp:789
#50 0x00007feac33fd35b in xmlrpc_c::processXmlrpcCall (envP=0x7feabd902b80, arg=0x249e100, 
    callXml=0x7fe8e8001650 "<?xml version=\"1.0\" encoding=\"ISO-8859-1\"?>\r\n<methodCall><methodName>daq.shutdown</methodName><params></params></methodCall>", callXmlLen=124, abyssSessionP=0x7feabd902c30, responseXmlPP=0x7feabd902b78) at server_abyss.cpp:381
---Type <return> to continue, or q <return> to quit---
#51 0x00007feac31f55e7 in processCall (trace=<optimized out>, accessControl=..., wantChunk=false, xmlProcessorArg=0x249e100, 
    xmlProcessor=0x7feac33fd2b0 <xmlrpc_c::processXmlrpcCall(xmlrpc_env*, void*, char const*, size_t, TSession*, xmlrpc_mem_block**)>, 
    contentSize=<optimized out>, abyssSessionP=0x7feabd902c30) at abyss_handler.c:392
#52 handleXmlRpcCallReq (requestInfoP=<optimized out>, accessControl=..., wantChunk=<optimized out>, xmlProcessorArg=<optimized out>, 
    xmlProcessor=0x7feac33fd2b0 <xmlrpc_c::processXmlrpcCall(xmlrpc_env*, void*, char const*, size_t, TSession*, xmlrpc_mem_block**)>, 
    abyssSessionP=0x7feabd902c30) at abyss_handler.c:509
#53 xmlrpc_handleIfXmlrpcReq (handlerArg=<optimized out>, abyssSessionP=0x7feabd902c30, handledP=<optimized out>) at abyss_handler.c:573
#54 0x00007feac29ba39f in runUserHandler (srvP=0x249e1a0, sessionP=0x7feabd902c30) at server.c:631
#55 processRequestFromClient (connectionP=connectionP@entry=0x249e580, lastReqOnConn=lastReqOnConn@entry=false, timeout=<optimized out>, 
    keepAliveP=keepAliveP@entry=0x7feabd902d84) at server.c:724
#56 0x00007feac29ba5d0 in serverFunc (userHandle=0x249e580) at server.c:796
#57 0x00007feac29b444f in connJob (userHandle=0x249e580) at conn.c:39
#58 0x00007feac29bded2 in execute (arg=0x249e000) at thread_pthread.c:59
#59 0x00007feacd87ce25 in start_thread () from /lib64/libpthread.so.0
#60 0x00007feacb0ab34d in clone () from /lib64/libc.so.6

#4 Updated by Kurt Biery 12 months ago

There may have been fewer crashes during this time frame because people were running the system less often because the timing system was being upgraded. Still, it's encouraging that there were only 3 EB core files. There were many more art core files. I only examined a few of those, and they seem to be due to SIGABRT, so I was guessing that they were forcible shutdowns when end-run took too long.



Also available in: Atom PDF