Project

General

Profile

Feature #5182

Need to be able to merge two currently unmerge-able files.

Added by Andrei Gaponenko about 7 years ago. Updated over 3 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Kyle Knoepfel
Category:
I/O
Target version:
2.08.00
Start date:
01/09/2014
Due date:
% Done:

100%

Estimated time:
40.00 h
Spent time:
10.00 h
Scope:
Internal
Experiment:
Mu2e
SSI Package:
art
Duration:

Description

Hello,

I get

cet::exception caught in art
---- UnimplementedFeature BEGIN
  Cannot merge file '25.root' due to a branch mismatch.
  Contact the framework group.
  cet::exception caught in EventProcessor and rethrown
---- UnimplementedFeature END

when trying to merge files. The data products that are retained
have the same types/module labels/instance names/process names.
The exception above is thrown even after all metadata are supposed
to be dropped.

A simple way to reproduce is to use the attached fcl file and do

   mu2e -c rewrite.fcl /mu2e/data/tdr/beam/g4s3p2/tdr.beam.g4s3.ds23.1009a_1025a_1223a.13872813/good/00000/beamFlashOnMothers.root
   mv rewrite.root 09.root
   mu2e -c rewrite.fcl /mu2e/data/tdr/beam/g4s3p2/tdr.beam.g4s3.ds23.1025a_1025a_1223a.14019207/good/00000/beamFlashOnMothers.root
   mv rewrite.root 25.root
   mu2e -c rewrite.fcl 09.root 25.root

The last command fails with the exception above.
Andrei

rewrite.fcl (273 Bytes) rewrite.fcl Andrei Gaponenko, 01/09/2014 06:04 PM

Associated revisions

Revision 60f58d99 (diff)
Added by Christopher Green almost 7 years ago

Event-shape tests for issue #5182, and some splitting of CMakeLists.txt

History

#1 Updated by Christopher Green about 7 years ago

  • Category set to I/O
  • Status changed from New to Accepted
  • Target version set to 493
  • Estimated time set to 16.00 h
  • SSI Package art added
  • SSI Package deleted ()

Time estimate is for a preliminary analysis only. A better estimate of the time to fix the problem is not possible at this time.

#2 Updated by Christopher Green almost 7 years ago

  • Status changed from Accepted to Assigned
  • Assignee set to Christopher Green

I have started to investigate this issue, and have succeeded in building the Mu2e software against a private (and therefore changeable) version of art in debug mode, and duplicating the issue. Tuesday afternoon I hope to be able to investigate exactly what is happening.

#3 Updated by Christopher Green almost 7 years ago

  • % Done changed from 0 to 20
  • Estimated time changed from 16.00 h to 40.00 h

This problem is now understood. In order to continue to detect files which really are incompatible, we must be very careful in proceeding with a fix, and several new tests must be constructed carefully in order to avoid the introduction of new problems.

Basically, "Drop metadata for dropped products" does not deal with the list of lists of BranchID for each process, which is the basis for assignment of ProductID for Ptr in products. In order to remedy this issue, we must carefully check that any product depending on a product with dropped metadata is also dropped in order to be sure that there can be no inconsistencies with references to missing or incorrect products in a particular event.

#4 Updated by Christopher Green almost 7 years ago

  • Tracker changed from Bug to Feature
  • Subject changed from file merge UnimplementedFeature to Need to be able to merge two currently unmerge-able files.
  • Status changed from Assigned to Accepted
  • Assignee deleted (Christopher Green)
  • Priority changed from High to Normal

Tests for current inconsistency checking implemented with 9febe38.

Unfortunately, as we discussed, the aspects of your two files that make them unmerge-able really do make them unmerge-able, in that the histories of the single product in the two files are different, rendering safe use of Ptr in the current scheme impossible. I understand you are not using Ptr here, but we are not in a position to detect the lack of use and relax the rules within the current scheme for Ptr.

Implementation of this feature will have to wait for the transition and rationalization of the metadata to SQLite. Please contact us if you need help with a workaround in the meantime.

#5 Updated by Christopher Green almost 7 years ago

  • Target version changed from 493 to 1.09.00

#6 Updated by Christopher Green almost 7 years ago

  • Target version changed from 1.09.00 to 521

#7 Updated by Rob Kutschke almost 7 years ago

Can we have a clear description of the rules that must be followed in order for files to be merge-able. I am looking for an explanation pitched at an expert or a near-expert.

#8 Updated by Kyle Knoepfel over 3 years ago

  • Status changed from Accepted to Resolved
  • Assignee set to Kyle Knoepfel
  • Target version deleted (521)
  • % Done changed from 20 to 100

This issue has been resolved as a consequence of issue #16602. For art versions 2.07 and older, the rules about concatenating input files can be found here.

#9 Updated by Kyle Knoepfel over 3 years ago

  • Target version set to 2.08.00

#10 Updated by Kyle Knoepfel over 3 years ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF