Mixing input goes into infinite loop
When reading a secondary stream input (mixing) with multiple files, the MixHelper can go into an infinite loop if the number of requested events exceeds the number in the current file. MixHelper gives the following message (repeating 'forever'):
%MSG-w MixingInputWrap: MixBackgroundFrames:flashMixerTrkCal@ 15-Nov-2018 16:27:49 CST run: 1002 subRun: 0 event: 4
Wrapping around to initial input file for mixing after 4288 secondary events read.
Note this can occur in any read mode, including 'sequential'.
By contrast, if the same set of mixing input files are first concatenated using 'artcat' and then passed to the mixing input as a single file, the mixing job works as expected. Naively I expect the mixing to behave the same regardless of how the inputs are divided into files, preferably for all read modes, but certainly for sequential. Also the fact that the failure manifests as an infinite loop and not an exception tells me this is unexpected behavior (ie a bug).
A pair of example jobs which demonstrate the problem can be found in:
/mu2e/app/users/brownd/debug/Offline/debug/cnf.mu2e.debug.MDC2018e.001002_00000000.fcl (15 files as mixing input)
/mu2e/app/users/brownd/debug/Offline/debugcat/cnf.mu2e.debugcat.MDC2018e.001002_00000000.fcl (single file made of same 15 files pre-concatenated by artcat)
#3 Updated by Kyle Knoepfel over 2 years ago
- Status changed from Assigned to Feedback
The problem is understood. Due to complications related to ROOT-file management, the current product-mixing system does not support reading secondary events across multiple secondary input files for a given event. When a file is opened, the mixing system checks if the number of secondary events requested exceeds the number of events in a secondary input file. If so, then the system moves to the next secondary input file. If file-wrapping is enabled, then the system will go back to the first secondary input file, and the process repeats ad infinitum, as observed.
To support mixing events from multiple secondary input files for a given event requires re-architecting some of the system. There is no fundamental reason why it cannot be done; however, we are unlikely to have a full solution by the time art 2.11.05 is desired. At the very least, we have implemented a check to ensure that infinite file-wrapping does not occur. If no file can be found that has the requisite number of events and wrapping has been enabled, then the following exception will be printed to the screen:
... Begin processing the 5th record. run: 1002 subRun: 0 event: 5 at 16-Nov-2018 13:34:18 CST %MSG-w MixingInputWrap: MixBackgroundFrames:flashMixerTrkCal@BeginModule 16-Nov-2018 13:34:42 CST run: 1002 subRun: 0 event: 6 Wrapping around to initial input file for mixing after 7427 secondary events read. %MSG ... %MSG-s ArtException: PostEndJob 16-Nov-2018 13:34:42 CST ModuleEndJob cet::exception caught in art ---- OtherArt BEGIN ---- EventProcessorFailure BEGIN EventProcessor: an exception occurred during current event processing ---- ScheduleExecutionFailure BEGIN Path: ProcessingStopped. ---- UnimplementedFeature BEGIN An error occurred while preparing product-mixing for the current event. The number of requested secondaries (3469) exceeds the number of events in any of the files specified for product mixing. For a read mode of 'SEQUENTIAL', the framework does not currently allow product-mixing to span multiple secondary input files for a given event. Please contact email@example.com for more information. cet::exception going through module MixBackgroundFrames/flashMixerTrkCal run: 1002 subRun: 0 event: 6 ---- UnimplementedFeature END Exception going through path TriggerPath ---- ScheduleExecutionFailure END ---- EventProcessorFailure END ---- OtherArt END %MSG Art has completed and will exit with status 1.
We propose that for art 2.11.05, the above check/exception be introduced (commit art:cf001eb), which will avoid the infinite loop. We would then ask that a separate feature request be made that allows a mix operation for an event to span multiple secondary input files. Is that acceptable?