Project

General

Profile

Feature #24429

Add the ability to print stack traces in ExceptionHandler

Added by Eric Flumerfelt about 2 months ago. Updated 3 days ago.

Status:
Reviewed
Priority:
Normal
Category:
-
Target version:
-
Start date:
05/18/2020
Due date:
% Done:

100%

Estimated time:
Experiment:
SBND
Co-Assignees:
Duration:

Description

Gennadiy has a utility for printing stack traces from when exceptions are thrown. We should incorporate this code into the artdaq::ExceptionHandler class so that we transparently gain this functionality across the system.

History

#1 Updated by Gennadiy Lukhanin 22 days ago

  • % Done changed from 0 to 90
  • Status changed from New to Work in progress

Checkout the feature/24429_print_stack_traces branch, export USE_EXCEPTIONSTACKTRACE=1, and rebuild. Misconfigure your DAQ system so exceptions are thrown and review the debug level messages from the ExceptionHandler class.

  4 1592507878910915 214553 214553   2              ExceptionHandler dbg .
  5 1592507878910912 214553 214553   2              ExceptionHandler dbg . #1 /daq/software/products/fhiclcpp/v4_11_00/slf7.x86_64.e19.py2.prof/lib/libfhiclcpp.so : +0x61e4a + 0x0
  6 1592507878910912 214553 214553   2              ExceptionHandler dbg . #2 /daq/software/products/fhiclcpp/v4_11_00/slf7.x86_64.e19.py2.prof/lib/libfhiclcpp.so : fhicl::parse_document(std::istream&, cet::filepath_maker&, fhicl::interme
diate_table&) + 0x2d
  7 1592507878910912 214553 214553   2              ExceptionHandler dbg . #3 /daq/software/products/fhiclcpp/v4_11_00/slf7.x86_64.e19.py2.prof/lib/libfhiclcpp.so : fhicl::make_ParameterSet(std::__cxx11::basic_string<char, std::char_trait
s<char>, std::allocator<char
  8 1592507878910911 214553 214553   2              ExceptionHandler dbg . #4 artdaqDriver : LoadParameterSet(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) + 0x7f
  9 1592507878910911 214553 214553   2              ExceptionHandler dbg . #5 artdaqDriver :  + 0x0
 10 1592507878910910 214553 214553   2              ExceptionHandler dbg . #6 artdaqDriver : main + 0x5b
 11 1592507878910901 214553 214553   2              ExceptionHandler dbg . Stack Trace:
 12 1592507878910900 214553 214553   2              ExceptionHandler dbg . Caught a "cet::coded_exception<fhicl::error, &fhicl::detail::translate[abi:cxx11]>" exception.
 13 1592507878908666 214553 214553   2              ExceptionHandler err . cet::exception object caught:---- BernCRTZMQDatafebctl Received no reply from febdrv after waiting for 5 seconds BEGIN;  ;---- BernCRTZMQDatafebctl Received no rep

Important note: All projects should be compiled with the -rdynamic option to allow backtraces. Add the following snippet to the top CMakeLists.txt of all projects.
Additional info: https://www.gnu.org/software/libc/manual/html_node/Backtraces.html

#exception stack trace
if(DEFINED ENV{USE_EXCEPTIONSTACKTRACE})
        if($ENV{USE_EXCEPTIONSTACKTRACE} GREATER 0)
                message("!!! Exception StackTrace Activated!!!")
                add_definitions(-D EXCEPTIONSTACKTRACE)
                set( EXCEPTIONSTACKTRACE_LIB  artdaq-core_Utilities_ExceptionStackTrace )
                cet_add_compiler_flags( -rdynamic )
        endif()
endif()

#2 Updated by Gennadiy Lukhanin 15 days ago

Ran code thru clang-tidy .... No complaints ....

#3 Updated by Gennadiy Lukhanin 15 days ago

  • % Done changed from 90 to 100
  • Status changed from Work in progress to Resolved

Tested with artdaq_demo on mu2edaq13.

I produced this error by temporarily removing the libartdaq-demo_Generators_ToySimulator_generator.so from the product lib directory and starting the demo.

$ tshow |grep ExceptionHandler
585 1593109534388860  79563 100952  21                        ExceptionHandler dbg .
586 1593109534388857  79563 100952  21                        ExceptionHandler dbg . #1 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc++.so.8 : +0x6fc6 + 0x0
587 1593109534388857  79563 100952  21                        ExceptionHandler dbg . #2 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande
588 1593109534388857  79563 100952  21                        ExceptionHandler dbg . #3 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande
589 1593109534388856  79563 100952  21                        ExceptionHandler dbg . #4 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/lib/libartdaq_ExternalComms_xmlrpc_commande
590 1593109534388856  79563 100952  21                        ExceptionHandler dbg . #5 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande
591 1593109534388852  79563 100952  21                        ExceptionHandler dbg . #6 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande                                                                                                                                                                                                  592 1593109534388851  79563 100952  21                        ExceptionHandler dbg . #7 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server.so.3 : xmlrpc_dispatchCall + 0x92
593 1593109534388851  79563 100952  21                        ExceptionHandler dbg . #8 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server.so.3 : xmlrpc_registry_process_call2
+ 0xd2
594 1593109534388851  79563 100952  21                        ExceptionHandler dbg . #9 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server++.so.8 : xmlrpc_c::registry::processC
all(std::__cxx11::basic_string<char, std:
595 1593109534388850  79563 100952  21                        ExceptionHandler dbg . #10 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server_abyss++.so.8 : xmlrpc_c::serverAbyss
_impl::processCall(std::__cxx11::basic_st
596 1593109534388850  79563 100952  21                        ExceptionHandler dbg . #11 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server_abyss++.so.8 : +0x6db3 + 0x0
597 1593109534388849  79563 100952  21                        ExceptionHandler dbg . #12 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server_abyss.so.3 : xmlrpc_handleIfXmlrpcRe
q + 0x568
598 1593109534388849  79563 100952  21                        ExceptionHandler dbg . #13 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0xce89 + 0x0
599 1593109534388849  79563 100952  21                        ExceptionHandler dbg . #14 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0xcfb3 + 0x0                 600 1593109534388848  79563 100952  21                        ExceptionHandler dbg . #15 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0x837f + 0x0
601 1593109534388848  79563 100952  21                        ExceptionHandler dbg . #16 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0x14162 + 0x0
602 1593109534388847  79563 100952  21                        ExceptionHandler dbg . Stack Trace:
603 1593109534388836  79563 100952  21                        ExceptionHandler dbg . Caught a "xmlrpc_c::fault" exception.
08 1593109534388340  79564 100951  29                        ExceptionHandler dbg .                                                                                                                                                       609 1593109534388338  79564 100951  29                        ExceptionHandler dbg . #1 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc++.so.8 : +0x6fc6 + 0x0
610 1593109534388337  79564 100951  29                        ExceptionHandler dbg . #2 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/lib/libartdaq_ExternalComms_xmlrpc_commande
611 1593109534388337  79564 100951  29                        ExceptionHandler dbg . #3 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande
612 1593109534388337  79564 100951  29                        ExceptionHandler dbg . #4 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande                                                                                                                                                                                                  613 1593109534388336  79564 100951  29                        ExceptionHandler dbg . #5 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande                                                                                                                                                                                                  614 1593109534388336  79564 100951  29                        ExceptionHandler dbg . #6 /home/lukhanin/sbnd_artdaq/work_dirs/lukhanin-sbnd-v0_06_02-developwithartdaq/products_experimental/artdaq/v3_08_00/slf7.x86_64.e19.py2.s94.prof/li
b/libartdaq_ExternalComms_xmlrpc_commande                                                                                                                                                                                                  615 1593109534388335  79564 100951  29                        ExceptionHandler dbg . #7 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server.so.3 : xmlrpc_dispatchCall + 0x92
616 1593109534388325  79564 100951  29                        ExceptionHandler dbg . #8 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server.so.3 : xmlrpc_registry_process_call2
+ 0xd2
617 1593109534388325  79564 100951  29                        ExceptionHandler dbg . #9 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server++.so.8 : xmlrpc_c::registry::processC
all(std::__cxx11::basic_string<char, std:
618 1593109534388325  79564 100951  29                        ExceptionHandler dbg . #10 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server_abyss++.so.8 : xmlrpc_c::serverAbyss
_impl::processCall(std::__cxx11::basic_st
619 1593109534388324  79564 100951  29                        ExceptionHandler dbg . #11 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server_abyss++.so.8 : +0x6db3 + 0x0        620 1593109534388324  79564 100951  29                        ExceptionHandler dbg . #12 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_server_abyss.so.3 : xmlrpc_handleIfXmlrpcRe
q + 0x568
621 1593109534388323  79564 100951  29                        ExceptionHandler dbg . #13 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0xce89 + 0x0
622 1593109534388323  79564 100951  29                        ExceptionHandler dbg . #14 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0xcfb3 + 0x0
623 1593109534388323  79564 100951  29                        ExceptionHandler dbg . #15 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0x837f + 0x0
624 1593109534388322  79564 100951  29                        ExceptionHandler dbg . #16 /scratch/lukhanin/sbndaqbuilds/products/xmlrpc_c/v1_51_03a/Linux64bit+3.10-2.17-e19-prof/lib/libxmlrpc_abyss.so.3 : +0x14162 + 0x0
625 1593109534388322  79564 100951  29                        ExceptionHandler dbg . Stack Trace:
626 1593109534388321  79564 100951  29                        ExceptionHandler dbg . Caught a "xmlrpc_c::fault" exception.
627 1593109534383342  79563 100952  45                        ExceptionHandler err . cet::exception object caught:---- Configuration BEGIN;  UnknownPlugin;  ---- Configuration BEGIN;    Library specification "ToySimulator" does not cor
respond to any library in CET_PLUGIN_PATH
628 1593109534383221  79563 100952  45                        ExceptionHandler err . Exception thrown during initialization of fragment generator of type "ToySimulator" 
629 1593109534382674  79564 100951  29                        ExceptionHandler err . cet::exception object caught:---- Configuration BEGIN;  UnknownPlugin;  ---- Configuration BEGIN;    Library specification "ToySimulator" does not cor
respond to any library in CET_PLUGIN_PATH
630 1593109534382559  79564 100951  29                        ExceptionHandler err . Exception thrown during initialization of fragment generator of type "ToySimulator" 

#4 Updated by Eric Flumerfelt 3 days ago

  • Status changed from Resolved to Reviewed

I have a made a few minor edits and am now satisfied with this functionality.



Also available in: Atom PDF