Project

General

Profile

Bug #23298

Reading a protoDUNE raw data file that has RawFragmentHeaderV0 in it with newer software causes the art program to crash

Added by Kurt Biery 27 days ago. Updated 23 days ago.

Status:
Closed
Priority:
Normal
Category:
Known Issues
Target version:
Start date:
09/19/2019
Due date:
% Done:

100%

Estimated time:
(Total: 0.50 h)
Experiment:
-
Co-Assignees:
Duration:

Description

I'm not yet sure if this is a bug or user error or what, but I've noticed that if try to look at "older" raw data files at protoDUNE with "newer" artdaq software, the art process crashes.

To help debug this, I've copied such an "older" raw data file to
  • mu2edaq:/home/biery/pdune/np04_raw_run009673_0001_dl1.root.copied

To reproduce the problem, I need to use the "verboseRawEventDump.fcl" config.

Here is what I see when I've set up a software area that uses the latest/greatest artdaq/artdaq_core code:

[biery@mu2edaq01 pdune]$ art -c verboseRawEventDump.fcl np04_raw_run009673_0001_dl1.root.copied
%MSG-i MF_INIT_OK: Early 19-Sep-2019 09:36:45 CDT JobSetup
Messagelogger initialization complete.
%MSG
%MSG-i PathConfiguration: Early 19-Sep-2019 09:36:45 CDT JobSetup
Multiple end paths have been combined into one end path,
"end_path" since order is irrelevant.

%MSG
19-Sep-2019 09:36:45 CDT Initiating request to open input file "np04_raw_run009673_0001_dl1.root.copied"
19-Sep-2019 09:36:45 CDT Opened input file "np04_raw_run009673_0001_dl1.root.copied"

Warning in artdaq::PrintBuildInfo module: Run 9673 appears not to have found product instance "ArtdaqDemo" of module "BuildInfo"

Begin processing the 1st record. run: 9673 subRun: 1 event: 1 at 19-Sep-2019 09:36:45 CDT
  • Start of EventDump for event 1 ***
    ContainerCPUHITS fragments:
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment ID 368 has type 232, timestamp 78444909647414821, and sizeBytes 1208978560 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647414821, and sizeBytes 0
    Upgrading RawFragmentHeaderV1 (const)
    2) fragment ID 363 has type 232, timestamp 78444909647414821, and sizeBytes 1209054336 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647414821, and sizeBytes 0
    Upgrading RawFragmentHeaderV1 (const)
    3) fragment ID 351 has type 232, timestamp 78444909647414821, and sizeBytes 208 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 152
    Upgrading RawFragmentHeaderV1 (const)
    4) fragment ID 361 has type 232, timestamp 78444909647414821, and sizeBytes 0 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647414821, and sizeBytes 0
    Upgrading RawFragmentHeaderV1 (const)
    5) fragment ID 353 has type 232, timestamp 78444909647414821, and sizeBytes 1209070464 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647414821, and sizeBytes 0
    Upgrading RawFragmentHeaderV1 (const)
    6) fragment ID 354 has type 232, timestamp 78444909647414821, and sizeBytes 1209079424 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 1208
    Upgrading RawFragmentHeaderV1 (const)
    7) fragment ID 359 has type 232, timestamp 78444909647414821, and sizeBytes 128 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647414821, and sizeBytes 0
    Upgrading RawFragmentHeaderV1 (const)
    8) fragment ID 358 has type 232, timestamp 78444909647414821, and sizeBytes 0 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647414821, and sizeBytes 0
    Upgrading RawFragmentHeaderV1 (const)
    9) fragment ID 367 has type 232, timestamp 78444909647414821, and sizeBytes 2976 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 2920
    Upgrading RawFragmentHeaderV1 (const)
    10) fragment ID 355 has type 232, timestamp 78444909647414821, and sizeBytes 2512 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 2456
    Upgrading RawFragmentHeaderV1 (const)
    11) fragment ID 350 has type 232, timestamp 78444909647414821, and sizeBytes 2096 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 2040
    Upgrading RawFragmentHeaderV1 (const)
    12) fragment ID 356 has type 232, timestamp 78444909647414821, and sizeBytes 2000 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 1944
    Upgrading RawFragmentHeaderV1 (const)
    13) fragment ID 366 has type 232, timestamp 78444909647414821, and sizeBytes 1102043776 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 280
    Upgrading RawFragmentHeaderV1 (const)
    14) fragment ID 362 has type 232, timestamp 78444909647414821, and sizeBytes 3824 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 3768
    Upgrading RawFragmentHeaderV1 (const)
    15) fragment ID 360 has type 232, timestamp 78444909647414821, and sizeBytes 2752 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 2696
    Upgrading RawFragmentHeaderV1 (const)
    16) fragment ID 369 has type 232, timestamp 78444909647414821, and sizeBytes 1102446208 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 2456
    Upgrading RawFragmentHeaderV1 (const)
    17) fragment ID 365 has type 232, timestamp 78444909647414821, and sizeBytes 6688 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 6632
    Upgrading RawFragmentHeaderV1 (const)
    18) fragment ID 364 has type 232, timestamp 78444909647414821, and sizeBytes 1102348800 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647414821, and sizeBytes 1102353792
    Upgrading RawFragmentHeaderV1 (const)
    19) fragment ID 352 has type 232, timestamp 78444909647414821, and sizeBytes 1296 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 1240
    Upgrading RawFragmentHeaderV1 (const)
    20) fragment ID 357 has type 232, timestamp 78444909647414821, and sizeBytes 4704 (contents: type = 11, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 11, timestamp 78444909647414821, and sizeBytes 4648
    ContainerCTB fragments:
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment ID 0 has type 232, timestamp 78444909647414821, and sizeBytes 1209063296 (contents: type = 10, count = 4, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment type 0, timestamp 78444909647076019, and sizeBytes 1208974592
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    2) fragment type 0, timestamp 78444909647201019, and sizeBytes 1209065344
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    3) fragment type 0, timestamp 78444909647326019, and sizeBytes 1102466944
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    Upgrading RawFragmentHeaderV1 (const)
    4) fragment type 0, timestamp 78444909647451019, and sizeBytes 1102958464
    ContainerFELIX fragments:
    Upgrading RawFragmentHeaderV1 (const)
    1) fragment ID 376 has type 232, timestamp 78444909647414821, and sizeBytes 20201160128 (contents: type = 8, count = 1, missing data = 0)
    Upgrading RawFragmentHeaderV1 (const)
    Segmentation fault (core dumped)

GDB seems to show a problem in Fragment::size():

Program received signal SIGSEGV, Segmentation fault.
0x00007fffe13cbd72 in artdaq::Fragment::size (this=<optimized out>)
at /home/biery/Sep03Demo/srcs/artdaq_core/artdaq-core/Data/Fragment.hh:845
845 return fragmentHeader()->word_count;

I'll keep poking around, but I wanted to let others know about this possible problem.


Subtasks

Feature #23299: Make Fragment upgrade messages less annoyingClosedEric Flumerfelt

Associated revisions

Revision 68dc6a80 (diff)
Added by Eric Flumerfelt 27 days ago

Add test cases for upgrading old versions of RawFragmentHeader to the
latest.

Part of the resolution of Issue #23298

Revision 98793d40 (diff)
Added by Eric Flumerfelt 27 days ago

Ensure that the hdr pointer is updated in
artdaq::Fragment::fragmentHeader() after vals_.insert is called.

Resolves #23298

History

#1 Updated by Eric Flumerfelt 27 days ago

  • Due date set to 09/19/2019

due to changes in a related task: #23299

#2 Updated by Eric Flumerfelt 27 days ago

I have reproduced the issue using e17:s67:debug and e17:s73:debug. Interestingly, e19:s82:prof does not have issues processing this file.

#3 Updated by Eric Flumerfelt 27 days ago

  • Assignee set to Eric Flumerfelt
  • Status changed from New to Resolved
  • Category set to Known Issues

I have traced this issue down to a pointer-invalidation issue affecting the const upgrade function called from artdaq::Fragment::fragmentHeader(). Resolution is on artdaq-core branch bugfix/23298_Fragment_UpgradeMethodInvalidatedPointer. I have also added test cases to Fragment_t which fail before the change and succeed after.

#4 Updated by Ron Rechenmacher 26 days ago

  • Status changed from Resolved to Reviewed

Code reviewed

#5 Updated by Eric Flumerfelt 23 days ago

  • Target version set to artdaq_core v3_05_03
  • Co-Assignees Ron Rechenmacher added

#6 Updated by Eric Flumerfelt 23 days ago

  • Status changed from Reviewed to Closed


Also available in: Atom PDF