Project

General

Profile

Bug #5423

Crashes in the Online Monitoring Aggregator related to the trigger monitoring module

Added by Kurt Biery over 5 years ago. Updated about 5 years ago.

Status:
New
Priority:
Normal
Assignee:
-
Target version:
Start date:
02/11/2014
Due date:
% Done:

0%

Estimated time:
12.00 h
Duration:

Description

In the early morning hours of 10-Feb (LNGS time), there were a number of crashes of the online monitoring Aggregator. Fortunately, core dumps were generated, and it looks like the problem is in the trigger monitoring module (MonitorTrigger_module.cc).

It seems like the problem happens when the horizontal scale of the trigger histogram is changed, but it is not obvious why that would be a problem, and attempts to reproduce the problem at the FNAL WH14NE teststand have so far been unsuccessful.

History

#1 Updated by Kurt Biery over 5 years ago

[dsfr1:1003:0]~/current/profile$ gdb AggregatorMain core.65103
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/&gt;...
Reading symbols from /home/daq/products/artdaq/v1_05_06/slf6.x86_64.e4.ib.prof/bin/AggregatorMain...done.
[New LWP 65114]
[New LWP 65105]
[New LWP 3306]
[New LWP 65103]
[New LWP 65107]
[New LWP 65113]

warning: Can't read pathname for load map: Input/output error.
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `AggregatorMain p 5461'.
Program terminated with signal 11, Segmentation fault.
#0 ds50::MonitorTrigger::analyze (this=0x1ad25d0, e=...) at /home/daq/current/ds50daq/ds50daq/online/ArtModules/MonitorTrigger_module.cc:94
94 x.second = std::unique_ptr<TH1F>(new TH1F("TriggerRate","Trigger Rate [Hz] ",y
>GetNbinsX () * 2, 0, y->GetBinLowEdge(y->GetNbinsX () + 1) * 2));
(gdb)
(gdb)
(gdb) where
#0 ds50::MonitorTrigger::analyze (this=0x1ad25d0, e=...) at /home/daq/current/ds50daq/ds50daq/online/ArtModules/MonitorTrigger_module.cc:94
#1 0x00007f57684613f7 in art::EDAnalyzer::doEvent (this=0x1ad25d0, ep=..., cpc=<optimized out>)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/Core/EDAnalyzer.cc:23
#2 0x00007f57686f1dcd in art::Worker::doWork<art::OccurrenceTraits<art::EventPrincipal, (art::BranchActionType)0> > (this=0x1ad2800, ep=...,
cpc=cpc@entry=0x7f5757dfb910) at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/Principal/Worker.h:221
#3 0x00007f57686f2b7e in runWorker<art::OccurrenceTraits<art::EventPrincipal, (art::BranchActionType)0> > (cpc=0x7f5757dfb910, ep=..., this=0x1ad3720)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/Core/WorkerInPath.h:79
#4 art::Path::processOneOccurrence<art::OccurrenceTraits<art::EventPrincipal, (art::BranchActionType)0> > (this=0x1ad2ed0, ep=...)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/Core/Path.h:166
#5 0x00007f57686f61ec in art::EndPathExecutor::processOneOccurrence<art::OccurrenceTraits<art::EventPrincipal, (art::BranchActionType)0> > (this=0x1ad0f90, ep=...)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/Core/EndPathExecutor.h:108
#6 0x00007f57686f66fa in art::EventProcessor::processOneOccurrence_<art::OccurrenceTraits<art::EventPrincipal, (art::BranchActionType)0> > (this=0x7f5758ae1ae0, p=...)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/EventProcessor/EventProcessor.h:235
#7 0x00007f57686dc479 in art::EventProcessor::processEvent (this=<optimized out>)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/EventProcessor/EventProcessor.cc:766
#8 0x00007f57686fea1b in statemachine::HandleEvent::readAndProcessEvent (this=this@entry=0x7f574ab61a50)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/EventProcessor/EPStates.cc:759
#9 0x00007f57686febe5 in statemachine::HandleEvent::HandleEvent (this=0x7f574ab61a50, ctx=...)
at /home/greenc/work/cet-is/test-products/art/v1_08_10/src/art/Framework/EventProcessor/EPStates.cc:726
#10 0x00007f5768707731 in boost::statechart::state<statemachine::HandleEvent, statemachine::HandleSubRuns, boost::mpl::list<mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na, mpl_::na>, (boost::statechart::history_mode)0>::shallow_construct (pContext=..., outermostContextBase=...)
at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/state.hpp:89
#11 0x00007f5768707a36 in deep_construct (outermostContextBase=..., pContext=...)
at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/state.hpp:79
#12 construct (outermostContextBase=..., pContext=...)
at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/detail/constructor.hpp:93
#13 transit_impl<statemachine::HandleEvent, statemachine::Machine, boost::statechart::detail::no_transition_function> (transitionAction=..., this=0x7f574ab61a50)
at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/simple_state.hpp:798
#14 transit<statemachine::HandleEvent> (this=0x7f574ab61a50)
at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/simple_state.hpp:314
#15 react_without_action (stt=...) at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/transition.hpp:38
#16 react (stt=...) at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/detail/reaction_dispatcher.hpp:47
#17 react (evt=..., eventType=<synthetic pointer>, stt=...)
at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/detail/reaction_dispatcher.hpp:92
#18 react (evt=..., eventType=<synthetic pointer>, stt=...)
at /home/greenc/work/cet-is/test-products/boost/v1_53_0/Linux64bit+2.6-2.12-e4-prof/include/boost/statechart/detail/reaction_dispatcher.hpp:109
---Type <return> to continue, or q <return> to quit---q
Quit

#2 Updated by Kurt Biery about 5 years ago

  • Target version set to KnownIssues

This happened again yesterday. No clues so far as to why it happened.



Also available in: Atom PDF