Inconsistent Process History Crashes¶
The so-called inconsistent process history crash produces an error message that looks like this:
---- MismatchedInputFiles BEGIN Cannot merge file '/storage/local/data1/condor/execute/dir_30979/no_xfer/ifdh_48258_1/PhysicsRun2016_2_27_21_20_2-0005187-00179_20160229T013909_ext_bnb_20160229T053058_merged_20160411T091804_reco1_20160412T080 058_reco2_20171025T155421_reco1_20171026T003046_reco2.root' due to inconsistent process histories: Process 1: Previous File File to merge ============================== 3887966044 3887966044 306682845 306682845 1246315526 1246315526 2033985239 2033985239 621916976 621916976 33066862 33066862 1414089015 1414089015 1259210377 1259210377 1970806703 1970806703 3474989124 3474989124 3844017340 3844017340 2662566863 2662566863 2127618975 2127618975 3793549530 3793549530 4197775376 4197775376 1917463921 1917463921 894316088 894316088 786784967 786784967 3991938222 3991938222 92763279 144616627 The BranchIDs above correspond to products that were created whenever the current input files were produced. The lists above must be identical per process. To determine which products these BranchIDs correspond to, rerun the corresponding processes that produced the input files, enabling full debug output for the message service. Then 'grep' the log file for messages with 'BranchID'. Contact the framework group for assistance. cet::exception caught in EventProcessor and rethrown ---- MismatchedInputFiles END %MSG Art has completed and will exit with status 65.
The root cause of this crash is incompatible reconstructed optical data products produced by different versions of the swizzler. The crash occurs even if you don't access the incompatible data products, or even if you drop them from the event. There are, however, at least three effecive workarounds, which are listed below.
- The simplest workaround is to use art version v2_08_04 or newer (uboonecode v06_55_00 or newer).
- Another workaround is to restrict the number of files per job to one (in
- If reading files from sam, it is possible to restrict files in a dataset to be compatible. If the original dataset definition is called
"mydef", define two new dataset definitions using the following commands.
$ samweb create-definition mydef1 "defname: mydef and defname: swizzle4" $ samweb create-definition mydef2 "defname: mydef and defname: swizzle5"
Here "swizzle4" and "swizzle5" are two predefined dataset definitions that consist of only files with compatible ancestry.