Project

General

Profile

Inconsistent Process History Crashes

The so-called inconsistent process history crash produces an error message that looks like this:

---- MismatchedInputFiles BEGIN
  Cannot merge file
 '/storage/local/data1/condor/execute/dir_30979/no_xfer/ifdh_48258_1/PhysicsRun2016_2_27_21_20_2-0005187-00179_20160229T013909_ext_bnb_20160229T053058_merged_20160411T091804_reco1_20160412T080
058_reco2_20171025T155421_reco1_20171026T003046_reco2.root' due to inconsistent process histories:

  Process 1:

    Previous File    File to merge
    ==============================
    3887966044       3887966044   
    306682845        306682845    
    1246315526       1246315526   
    2033985239       2033985239   
    621916976        621916976    
    33066862         33066862     
    1414089015       1414089015   
    1259210377       1259210377   
    1970806703       1970806703   
    3474989124       3474989124   
    3844017340       3844017340   
    2662566863       2662566863   
    2127618975       2127618975   
    3793549530       3793549530   
    4197775376       4197775376   
    1917463921       1917463921   
    894316088        894316088    
    786784967        786784967    
    3991938222       3991938222   
    92763279         144616627    

  The BranchIDs above correspond to products that were
  created whenever the current input files were produced.

  The lists above must be identical per process.

  To determine which products these BranchIDs correspond to,
  rerun the corresponding processes that produced the
  input files, enabling full debug output for the message
  service. Then 'grep' the log file for messages with 'BranchID'.

  Contact the framework group for assistance.
  cet::exception caught in EventProcessor and rethrown
---- MismatchedInputFiles END
%MSG
Art has completed and will exit with status 65.

The root cause of this crash is incompatible reconstructed optical data products produced by different versions of the swizzler. The crash occurs even if you don't access the incompatible data products, or even if you drop them from the event. There are, however, at least three effecive workarounds, which are listed below.

  1. The simplest workaround is to use art version v2_08_04 or newer (uboonecode v06_55_00 or newer).
  2. Another workaround is to restrict the number of files per job to one (in project.py use <maxfilesperjob>1</maxfilesperjob>).
  3. If reading files from sam, it is possible to restrict files in a dataset to be compatible. If the original dataset definition is called "mydef", define two new dataset definitions using the following commands.
    $ samweb create-definition mydef1 "defname: mydef and defname: swizzle4" 
    $ samweb create-definition mydef2 "defname: mydef and defname: swizzle5" 
    

    Here "swizzle4" and "swizzle5" are two predefined dataset definitions that consist of only files with compatible ancestry.