Project

General

Profile

Bug #21384

Mixing input goes into infinite loop

Added by David Brown 8 months ago. Updated 8 months ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Infrastructure
Target version:
Start date:
11/15/2018
Due date:
% Done:

100%

Estimated time:
Spent time:
Occurs In:
Scope:
Internal
Experiment:
Mu2e
SSI Package:
art
Duration:

Description

When reading a secondary stream input (mixing) with multiple files, the MixHelper can go into an infinite loop if the number of requested events exceeds the number in the current file. MixHelper gives the following message (repeating 'forever'):

%MSG-w MixingInputWrap: MixBackgroundFrames:flashMixerTrkCal@ 15-Nov-2018 16:27:49 CST run: 1002 subRun: 0 event: 4
Wrapping around to initial input file for mixing after 4288 secondary events read.

Note this can occur in any read mode, including 'sequential'.
By contrast, if the same set of mixing input files are first concatenated using 'artcat' and then passed to the mixing input as a single file, the mixing job works as expected. Naively I expect the mixing to behave the same regardless of how the inputs are divided into files, preferably for all read modes, but certainly for sequential. Also the fact that the failure manifests as an infinite loop and not an exception tells me this is unexpected behavior (ie a bug).

A pair of example jobs which demonstrate the problem can be found in:
/mu2e/app/users/brownd/debug/Offline/debug/cnf.mu2e.debug.MDC2018e.001002_00000000.fcl (15 files as mixing input)
vs
/mu2e/app/users/brownd/debug/Offline/debugcat/cnf.mu2e.debugcat.MDC2018e.001002_00000000.fcl (single file made of same 15 files pre-concatenated by artcat)


Related issues

Related to art - Feature #21412: Enable product-mixing to read from multiple secondary input files per primary eventAccepted2018-11-20

History

#1 Updated by Rob Kutschke 8 months ago

If this turns out to be straightforward to diagnose and fix, I would like to see it in art v2_11_05.

#2 Updated by Kyle Knoepfel 8 months ago

  • Category changed from Event Loop to Infrastructure
  • Status changed from New to Assigned
  • Assignee set to Kyle Knoepfel

I was able to reproduce the error. I am investigating.

#3 Updated by Kyle Knoepfel 8 months ago

  • Status changed from Assigned to Feedback

The problem is understood. Due to complications related to ROOT-file management, the current product-mixing system does not support reading secondary events across multiple secondary input files for a given event. When a file is opened, the mixing system checks if the number of secondary events requested exceeds the number of events in a secondary input file. If so, then the system moves to the next secondary input file. If file-wrapping is enabled, then the system will go back to the first secondary input file, and the process repeats ad infinitum, as observed.

To support mixing events from multiple secondary input files for a given event requires re-architecting some of the system. There is no fundamental reason why it cannot be done; however, we are unlikely to have a full solution by the time art 2.11.05 is desired. At the very least, we have implemented a check to ensure that infinite file-wrapping does not occur. If no file can be found that has the requisite number of events and wrapping has been enabled, then the following exception will be printed to the screen:

...

Begin processing the 5th record. run: 1002 subRun: 0 event: 5 at 16-Nov-2018 13:34:18 CST
%MSG-w MixingInputWrap:  MixBackgroundFrames:flashMixerTrkCal@BeginModule  16-Nov-2018 13:34:42 CST run: 1002 subRun: 0 event: 6
Wrapping around to initial input file for mixing after 7427 secondary events read.
%MSG

...

%MSG-s ArtException:  PostEndJob 16-Nov-2018 13:34:42 CST ModuleEndJob
cet::exception caught in art
---- OtherArt BEGIN
  ---- EventProcessorFailure BEGIN
    EventProcessor: an exception occurred during current event processing
    ---- ScheduleExecutionFailure BEGIN
      Path: ProcessingStopped.
      ---- UnimplementedFeature BEGIN
        An error occurred while preparing product-mixing for the current event.
        The number of requested secondaries (3469) exceeds the number of events in any
        of the files specified for product mixing.  For a read mode of 'SEQUENTIAL',
        the framework does not currently allow product-mixing to span multiple secondary
        input files for a given event.  Please contact artists@fnal.gov for more information.
        cet::exception going through module MixBackgroundFrames/flashMixerTrkCal run: 1002 subRun: 0 event: 6
      ---- UnimplementedFeature END
      Exception going through path TriggerPath
    ---- ScheduleExecutionFailure END
  ---- EventProcessorFailure END
---- OtherArt END
%MSG
Art has completed and will exit with status 1.

We propose that for art 2.11.05, the above check/exception be introduced (commit art:cf001eb), which will avoid the infinite loop. We would then ask that a separate feature request be made that allows a mix operation for an event to span multiple secondary input files. Is that acceptable?

#4 Updated by Kyle Knoepfel 8 months ago

  • Status changed from Feedback to Resolved
  • Target version set to 2.11.05
  • % Done changed from 0 to 100

Per discussion with Rob, art 2.11.05 will have the bug fix alluded to above.

#5 Updated by Kyle Knoepfel 8 months ago

  • Status changed from Resolved to Closed

#6 Updated by Kyle Knoepfel 8 months ago

  • Related to Feature #21412: Enable product-mixing to read from multiple secondary input files per primary event added


Also available in: Atom PDF