Project

General

Profile

Support #24917

How to combine art::Results from multiple files

Added by Brian Rebel about 2 months ago. Updated 15 days ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
-
Target version:
-
Start date:
09/05/2020
Due date:
% Done:

100%

Estimated time:
Scope:
Internal
Experiment:
NOvA
SSI Package:
art
Duration:

Description

I have a ResultsProducer that takes art::Results from multiple files and combines them into a single vector output to a new file. I drop all the original data products when writing the output ROOT file with this configuration

outputCommands: [ "drop *",
                    "keep *_*_*_CombineResults"]

The problem is that when I try to use the combined file I get this error at run time:

%MSG-s ArtException:  PostEndJob 05-Sep-2020 13:28:39 CDT ModuleEndJob
cet::exception caught in art
---- OtherArt BEGIN
  ---- EventRangeOverlap BEGIN
    Attempt to merge event ranges that both contain one or more of the same events
     Run: 1
      SubRun: 0 Event range: [1,2)  vs.
      SubRun: 0 Event range: [1,2)
  ---- EventRangeOverlap END
---- OtherArt END
%MSG
Art has completed and will exit with status 1.

There is nothing stored in the art::Event or art::SubRun, as only the art::Results has anything in it. Is there an obvious thing I am doing wrong?

My plugin is pretty simple so I am pasting it in here in case that helps you answer the question.

///////////////////////////////////////////////////////////////////////
/// \brief   Module to combine a set of results into a single file
///          currently only does one data product type at a time
/// \author  brebel@fnal.gov
///////////////////////////////////////////////////////////////////////
#include <vector>
#include <algorithm>

// Framework includes
#include "art/Framework/Core/ResultsProducer.h" 
#include "art/Framework/Core/ModuleMacros.h" 
#include "art/Framework/Principal/Results.h" 
#include "fhiclcpp/ParameterSet.h" 
#include "messagefacility/MessageLogger/MessageLogger.h" 

// NOvA includes
#include "CovarianceMatrixFit/dataProducts/PointResult.h" 
#include "CovarianceMatrixFit/dataProducts/GridPointResult.h" 
#include "CovarianceMatrixFit/dataProducts/FakeUniverse.h" 
#include "CovarianceMatrixFit/dataProducts/StaticFuncs.h" 
#include "CovarianceMatrixFit/utilities/ReadResults.h" 

namespace cmf {

  class CombineResults : public art::ResultsProducer {
  public:
    explicit CombineResults(fhicl::ParameterSet const& pset);
    virtual ~CombineResults();

    void readResults (art::Results const& r) override;
    void writeResults(art::Results      & r) override;
    void clear()                             override;

  private:

    template <class T> void ReadDataProducts(art::Results const& r,
                              std::vector<T>    & vec);

    template <class T> void WriteResultsVec(art::Results & r,
                                            std::vector<T> & resultsVec);

    std::string fInputDataProductLabel; ///< Name of the label for the input data products
    std::string fDataProductType;       ///< type of data product to combine

    // vectors of possible data products which could be read in - this approach
    // is not very elegant, it would be preferable to template these things somehow
    std::vector<cmf::FakeUniverse>  fFakeUniverseVec;
    std::vector<cmf::PointResult>   fPointResultVec;
    std::vector<cmf::OscParamPoint> fOscParamPointVec;
  };

  //......................................................................
  CombineResults::CombineResults(fhicl::ParameterSet const& pset)
  : fInputDataProductLabel(pset.get<std::string>("InputDataProductLabel"))
  , fDataProductType      (pset.get<std::string>("DataProductType"      ))
  {
    if     (fDataProductType == "FakeUniverse")  produces< std::vector<cmf::FakeUniverse>  >();
    else if(fDataProductType == "PointResult")   produces< std::vector<cmf::PointResult>   >();
    else if(fDataProductType == "OscParamPoint") produces< std::vector<cmf::OscParamPoint> >();
  }

  //......................................................................
  CombineResults::~CombineResults()
  {
  }

  //......................................................................
  // Method to clear out the collections of data products after the
  // writeResults method is called.
  void CombineResults::clear()
  {
  }

  //......................................................................
  template <class T> void CombineResults::ReadDataProducts(art::Results const& r,
                                            std::vector<T>    & vec)
  {
    if(cmf::readVectorResults(r, fInputDataProductLabel, vec, false)){
      LOG_VERBATIM("LibraryContourMaker")
          << "Valid handle to std::vector<cmf::" 
          << fDataProductType
          << "> " 
          << "objects found in " 
          << fInputDataProductLabel
          << " vector size " 
          << vec.size();
    }

  }

  //......................................................................
  void CombineResults::readResults(art::Results const& r)
  {
    if     (fDataProductType == "FakeUniverse" ) this->ReadDataProducts(r, fFakeUniverseVec );
    else if(fDataProductType == "PointResult"  ) this->ReadDataProducts(r, fPointResultVec  );
    else if(fDataProductType == "OscParamPoint") this->ReadDataProducts(r, fOscParamPointVec);
  }

  //......................................................................
  template <class T> void CombineResults::WriteResultsVec(art::Results & r,
                                                          std::vector<T> & resultsVec)
  {
    std::unique_ptr<std::vector<T>> combVec = std::make_unique<std::vector<T>>(resultsVec);
    r.put(std::move(combVec));
  }

  //......................................................................
  void CombineResults::writeResults(art::Results & r)
  {
    LOG_DEBUG("CombineResults")
        << "Start writeResults";

    if     (fDataProductType == "FakeUniverse" ) this->WriteResultsVec(r, fFakeUniverseVec );
    else if(fDataProductType == "PointResult"  ) this->WriteResultsVec(r, fPointResultVec  );
    else if(fDataProductType == "OscParamPoint") this->WriteResultsVec(r, fOscParamPointVec);

  } // end writeResults

  DEFINE_ART_RESULTS_PLUGIN(CombineResults)

} // end cmf namespace

and the job fhicl file is

#include "CMF_CombineResults.fcl" 
#include "messageservice.fcl" 

process_name: CombineResults

services:
{
 # Load the service that manages root files for histograms.
 message: @local::standard_info
}

# Otherwise ART holds onto all the objects until the end of the
# file and we run out of memory
services.scheduler.fileMode: FULLMERGE

source:
{
 module_type: RootInput
}

# Change the DATAPROD to whichever set of data products you are combining
outputs:
{
 combined:
 {
  module_type: RootOutput
  fileName:    "combined_output.root" 
  outputCommands: [ "drop *",
                    "keep *_*_*_CombineResults"]
  saveMemoryObjectThreshold: 104857600 # Drop ROOT buffers above 100MB
  results:
  {
   producers:
   {
    oscparampoint: @local::oscparampoint_combineresults
   }
   rpPath: [ oscparampoint ]
  }
 }
}

physics:
{
 stream1: [ combined ]
}

History

#1 Updated by Kyle Knoepfel about 2 months ago

  • Status changed from New to Feedback

Brian, this means that the input files at some point were generated with duplicate event numbers. art keeps track of the events that pertain to a given Run or SubRun so that users do not accidentally double count. The way to check for this is to type:

file_info_dumper --range-of-validity=full -s <input1>.root
file_info_dumper --range-of-validity=full -s <input2>.root
...

and that should tell you which file resulted from a concatenation with duplicate event IDs.

#2 Updated by Brian Rebel about 2 months ago

Hi Kyle

there are no event numbers for these files, except maybe a default? The only information placed in the file is at the art::Results level, so I am hoping there is a way to bypass these event range checks in that situation.

Thanks

#3 Updated by Kyle Knoepfel about 2 months ago

art keeps track of the events that went into creating the input files, even if all events in them have been dropped. At some point in the processing chain, two events were processed with the same number triplet (Run: 1, SubRun: 0, Event 1). In art 3, this check is not optional--there cannot be any indication of duplicate event IDs, otherwise the framework job will fail, as you have observed.

The solution to this problem is to reproduce the files with disjoint event numbers.

#4 Updated by Christopher Backhouse about 2 months ago

If these are NOvA MC files then we can indeed have duplicate run/subrun/event. This can happen if the same data run/subrun is sampled multiple times for generation. And since we generate much more MC exposure than data, that is pretty common. We can distinguish these files with the value of "cycle", but art doesn't know anything about that.

I think the above illustrates that art might want to consider allowing these duplicates, or providing an additional field to distinguish beyond the data-based run/subrun/event.

#5 Updated by Brian Rebel about 2 months ago

Hi Kyle

there is a disconnect between our assumptions. I don't think I used any event information in making these files. I started from a ResultsProducer that did not read in any previously made art files. In that case, shouldn't the run/subrun/event information be empty?

Thanks

Brian

#6 Updated by Kyle Knoepfel about 2 months ago

Can I see your configuration and output for the most upstream art process in this workflow?

#7 Updated by Brian Rebel about 2 months ago

An example fhicl configuration (there are 2499 of them) is below. You can see an example output file at

/pnfs/nova/scratch/users/brebel/predictions/2020/ThreeFlavor/cmf_prediction_library_result_0000_ThreeFlavor_0_28000_0_00230.root

# Produced from 'fhicl-dump' using:
#   Input  : cmf_predictionlibrary_191_ThreeFlavor_0_31000_0_00268.fcl
#   Policy : cet::filepath_maker
#   Path   : "FHICL_FILE_PATH" 

outputs: {
   prediction: {
      fileName: "cmf_predictionlibrary.root" 
      module_type: "RootOutput" 
      outputCommands: [
         "drop *",
         "keep *_*_*_PredictionLibraryMaker" 
      ]
      results: {
         producers: {
            library: {
               BinningToUse: {
                  nc_bin_edges: {
                     fd: {
                        high_edges: [
                           7.5e-1,
                           1,
                           1.25,
                           1.5,
                           1.75,
                           2,
                           2.25,
                           2.5,
                           2.75,
                           3,
                           3.25,
                           3.5,
                           3.75,
                           4,
                           4.25,
                           4.5,
                           4.75,
                           5,
                           5.5,
                           6,
                           6.5,
                           7,
                           7.5,
                           8,
                           8.5,
                           9,
                           9.5,
                           10,
                           11,
                           12,
                           13,
                           14,
                           15,
                           16,
                           17,
                           18,
                           19,
                           20
                        ]
                        low_edge: 5e-1
                     }
                     nd: {
                        high_edges: [
                           7.5e-1,
                           1,
                           1.25,
                           1.5,
                           1.75,
                           2,
                           2.25,
                           2.5,
                           2.75,
                           3,
                           3.25,
                           3.5,
                           3.75,
                           4,
                           4.25,
                           4.5,
                           4.75,
                           5,
                           5.5,
                           6,
                           6.5,
                           7,
                           7.5,
                           8,
                           8.5,
                           9,
                           9.5,
                           10,
                           11,
                           12,
                           13,
                           14,
                           15,
                           16,
                           17,
                           18,
                           19,
                           20
                        ]
                        low_edge: 5e-1
                     }
                  }
                  nue_bin_edges: {
                     fd: {
                        high_edges: [
                           1.5,
                           2,
                           2.5,
                           3,
                           3.5,
                           4
                        ]
                        low_edge: 1
                     }
                     nd: {
                        high_edges: [
                           1,
                           1.5,
                           2,
                           2.5,
                           3,
                           3.5,
                           4,
                           4.5
                        ]
                        low_edge: 0
                     }
                     peripheral: {
                        high_edges: [
                           10
                        ]
                        low_edge: 0
                     }
                  }
                  numu_bin_edges: {
                     fd: {
                        high_edges: [
                           7.5e-1,
                           1,
                           1.1,
                           1.2,
                           1.3,
                           1.4,
                           1.5,
                           1.6,
                           1.7,
                           1.8,
                           1.9,
                           2,
                           2.25,
                           2.5,
                           2.75,
                           3,
                           3.5,
                           4,
                           5
                        ]
                        low_edge: 0
                     }
                     nd: {
                        high_edges: [
                           7.5e-1,
                           1,
                           1.1,
                           1.2,
                           1.3,
                           1.4,
                           1.5,
                           1.6,
                           1.7,
                           1.8,
                           1.9,
                           2,
                           2.25,
                           2.5,
                           2.75,
                           3,
                           3.5,
                           4,
                           5
                        ]
                        low_edge: 0
                     }
                  }
               }
               FarDetEventListManipulator: {
                  CMFEventLabels: [
                     "/nova/ana/users/brebel/skimmed/cmf_eventlisttrees_2020Feb06.root" 
                  ]
                  DetectorsToUse: [
                     "Far" 
                  ]
                  EventCap: {
                     FarBeam: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarData: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarFluxSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarRockFluxSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarRockNonSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarTauSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     NearBeam: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     NearData: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                  }
                  FHCExposureMultiplier: 1
                  FakeExposure: [
                     {
                        Detector: "FarDet" 
                        EndRun: 17139
                        Exposure: 1.22735e20
                        Period: 1
                        StartRun: 0
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 19746
                        Exposure: 2.64398e20
                        Period: 2
                        StartRun: 17140
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 23419
                        Exposure: 3.84476e20
                        Period: 3
                        StartRun: 19747
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 25412
                        Exposure: 1.76205e20
                        Period: 5
                        StartRun: 24614
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 24613
                        Exposure: 6.5534e19
                        Period: 4
                        StartRun: 23420
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 28036
                        Exposure: 1.16769e21
                        Period: 6
                        StartRun: 25413
                     }
                  ]
                  LoadAllEventLists: false
                  MaxEventsPerTree: 175000
                  MaxNCEnergyForAnalysis: 20
                  MaxNuEEnergyForAnalysis: 5
                  MaxNuMuEnergyForAnalysis: 10
                  MinEventsPerTree: 100000
                  PeriodsToUse: [
                     {
                        Detector: "FarDet" 
                        EndRun: 17139
                        Exposure: 1.22735e20
                        Period: 1
                        StartRun: 0
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 19746
                        Exposure: 2.64398e20
                        Period: 2
                        StartRun: 17140
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 23419
                        Exposure: 3.84476e20
                        Period: 3
                        StartRun: 19747
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 25412
                        Exposure: 1.76205e20
                        Period: 5
                        StartRun: 24614
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 24613
                        Exposure: 6.5534e19
                        Period: 4
                        StartRun: 23420
                     },
                     {
                        Detector: "FarDet" 
                        EndRun: 28036
                        Exposure: 1.16769e21
                        Period: 6
                        StartRun: 25413
                     }
                  ]
                  Production: 4
                  RHCExposureMultiplier: 1
                  RandomSeed: 1.23456789e8
                  SumSubEpochs: true
                  TreeDirectories: [
                     "list" 
                  ]
                  UseEventId: false
               }
               GridPointPars: {
                  GridXVal: 5.90500015160318e-1
                  GridYVal: 2.68e-3
                  HiddenParameters: {
                     Th13: {
                        Divs: 10
                        LowerVal: 1.4e-1
                        ParName: "Th13" 
                        UpperVal: 1.5e-1
                     }
                     dCP: {
                        Divs: 30
                        LowerVal: 0
                        ParName: "dCP" 
                        UpperVal: 6.28318531
                     }
                  }
                  ParamX: "Th23" 
                  ParamY: "Dmsq32" 
                  UseTrig: true
               }
               NearDetEventListManipulator: {
                  CMFEventLabels: [
                     "/nova/ana/users/brebel/skimmed/cmf_eventlisttrees_2020Feb06.root" 
                  ]
                  DetectorsToUse: [
                     "Near" 
                  ]
                  EventCap: {
                     FarBeam: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarData: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarFluxSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarRockFluxSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarRockNonSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     FarTauSwap: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     NearBeam: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                     NearData: {
                        P1: 1
                        P10: 1
                        P101: 1
                        P102: 1
                        P2: 1
                        P3: 1
                        P4: 1
                        P5: 1
                        P6: 1
                        P7: 1
                        P8: 1
                        P9: 1
                     }
                  }
                  FHCExposureMultiplier: 1
                  FakeExposure: [
                     {
                        Detector: "NearDet" 
                        EndRun: 10407
                        Exposure: 1.22735e20
                        Period: 1
                        StartRun: 10377
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 11228
                        Exposure: 2.64398e20
                        Period: 2
                        StartRun: 10496
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 11628
                        Exposure: 3.84476e20
                        Period: 3
                        StartRun: 11229
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 12086
                        Exposure: 1.76205e20
                        Period: 5
                        StartRun: 11926
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 11925
                        Exposure: 6.5534e19
                        Period: 4
                        StartRun: 11632
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 12516
                        Exposure: 1.16769e21
                        Period: 6
                        StartRun: 12087
                     }
                  ]
                  LoadAllEventLists: false
                  MaxEventsPerTree: 175000
                  MaxNCEnergyForAnalysis: 20
                  MaxNuEEnergyForAnalysis: 5
                  MaxNuMuEnergyForAnalysis: 10
                  MinEventsPerTree: 100000
                  PeriodsToUse: [
                     {
                        Detector: "NearDet" 
                        EndRun: 10407
                        Exposure: 1.22735e20
                        Period: 1
                        StartRun: 10377
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 11228
                        Exposure: 2.64398e20
                        Period: 2
                        StartRun: 10496
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 11628
                        Exposure: 3.84476e20
                        Period: 3
                        StartRun: 11229
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 12086
                        Exposure: 1.76205e20
                        Period: 5
                        StartRun: 11926
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 11925
                        Exposure: 6.5534e19
                        Period: 4
                        StartRun: 11632
                     },
                     {
                        Detector: "NearDet" 
                        EndRun: 12516
                        Exposure: 1.16769e21
                        Period: 6
                        StartRun: 12087
                     }
                  ]
                  Production: 4
                  RHCExposureMultiplier: 1
                  RandomSeed: 1.23456789e8
                  SumSubEpochs: true
                  TreeDirectories: [
                     "list" 
                  ]
                  UseEventId: false
               }
               ParametersToUse: {
                  OscillationParameters: {
                     Dmsq21: {
                        CentralValue: 7.53e-5
                        Constraint: true
                        Detector: "BOTH" 
                        Fixed: true
                        HiVal: 1e-3
                        LoVal: 0
                        Nuisance: false
                        Parameter: "Dmsq21" 
                        Sigma: 2.6e-6
                        Value: 7.53e-5
                        Variance: 2.6e-6
                     }
                     Dmsq32: {
                        CentralValue: 2.51e-3
                        Constraint: false
                        Detector: "BOTH" 
                        Fixed: true
                        HiVal: 5e-3
                        LoVal: -5e-3
                        Nuisance: true
                        Parameter: "Dmsq32" 
                        Sigma: 5e-4
                        Value: 2.51e-3
                        Variance: 6e-5
                     }
                     Rho: {
                        CentralValue: 0
                        Constraint: false
                        Detector: "BOTH" 
                        Fixed: true
                        HiVal: 0
                        LoVal: 0
                        Nuisance: false
                        Parameter: "Rho" 
                        Sigma: 1e-6
                        Value: 2.84
                        Variance: 0
                     }
                     Th12: {
                        CentralValue: 5.87252e-1
                        Constraint: false
                        Detector: "BOTH" 
                        Fixed: true
                        HiVal: 3.14159265359
                        LoVal: 0
                        Nuisance: false
                        Parameter: "Th12" 
                        Sigma: 1.8e-2
                        Value: 5.87252e-1
                        Variance: 1.8e-2
                     }
                     Th13: {
                        CentralValue: 1.45426e-1
                        Constraint: false
                        Detector: "BOTH" 
                        Fixed: false
                        HiVal: 2e-1
                        LoVal: 1e-1
                        Nuisance: true
                        Parameter: "Th13" 
                        Sigma: 5e-3
                        Value: 1.45426e-1
                        Variance: 5e-3
                     }
                     Th23: {
                        CentralValue: 7.85398e-1
                        Constraint: false
                        Detector: "BOTH" 
                        Fixed: true
                        HiVal: 1.3
                        LoVal: 3e-1
                        Nuisance: true
                        Parameter: "Th23" 
                        Sigma: 5e-1
                        Value: 8.65744e-1
                        Variance: 0
                     }
                     dCP: {
                        CentralValue: 0
                        Constraint: true
                        Detector: "BOTH" 
                        Fixed: true
                        HiVal: 6.28318530718
                        LoVal: 0
                        Nuisance: false
                        Parameter: "dCP" 
                        Sigma: 3.1415
                        Value: 1.7e-1
                        Variance: 0
                     }
                     farL: {
                        CentralValue: 0
                        Constraint: false
                        Detector: "Far" 
                        Fixed: true
                        HiVal: 0
                        LoVal: 0
                        Nuisance: false
                        Parameter: "L" 
                        Sigma: 1e-6
                        Value: 810
                        Variance: 0
                     }
                     nearL: {
                        CentralValue: 0
                        Constraint: false
                        Detector: "Near" 
                        Fixed: true
                        HiVal: 0
                        LoVal: 0
                        Nuisance: false
                        Parameter: "L" 
                        Sigma: 1e-6
                        Value: 1
                        Variance: 0
                     }
                  }
                  SystematicParameters: {
                     xseccvwgt: {
                        CentralValue: 0
                        Detector: "BOTH" 
                        Fixed: true
                        Nuisance: false
                        Parameter: "XsecCVWgt" 
                        Sigma: 0
                        Value: 1
                        Variance: 0
                     }
                  }
               }
               SelectionsToUse: [
                  {
                     Beam: "FHC" 
                     Detector: "Far" 
                     Selections: [
                        "NuESel_LowPID",
                        "NuESel_HighPID",
                        "NuESel_Peripheral",
                        "NuMuSel" 
                     ]
                  },
                  {
                     Beam: "FHC" 
                     Detector: "Near" 
                     Selections: [
                        "NuESel_LowPID",
                        "NuESel_HighPID",
                        "NuMuSel" 
                     ]
                  },
                  {
                     Beam: "RHC" 
                     Detector: "Far" 
                     Selections: [
                        "NuESel_LowPID",
                        "NuESel_HighPID",
                        "NuESel_Peripheral",
                        "NuMuSel" 
                     ]
                  },
                  {
                     Beam: "RHC" 
                     Detector: "Near" 
                     Selections: [
                        "NuESel_LowPID",
                        "NuESel_HighPID",
                        "NuMuSel" 
                     ]
                  }
               ]
               ShifterAndWeighterParameters: {
                  CalibSystFile: "CovarianceMatrixFit/data/calibration/CalibrationSystematics2019.root" 
                  OscCalculatorType: "PMNS" 
                  TrueEOscCalc: true
               }
               plugin_type: "CMFPredictionLibraryMaker" 
            }
         }
         rpPath: [
            "library" 
         ]
      }
      saveMemoryObjectThreshold: 1.048576e8
   }
}
physics: {
   stream1: [
      "prediction" 
   ]
}
process_name: "PredictionLibraryMaker" 
services: {
   TFileService: {
      fileName: "cmf_predictionlibrary_hist.root" 
   }
   message: {
      destinations: {
         infomsg: {
            categories: {
               ELMUEFMDetector: {
                  limit: 1
               }
               default: {}
}
            threshold: "INFO" 
            type: "cout" 
}
}
}
   scheduler: {
      fileMode: "FULLMERGE" 
   }
}

#8 Updated by Kyle Knoepfel about 1 month ago

The issue is understood. When using EmptyEvent as the input source, it is necessary for at least one event to be processed to create an art/ROOT output file. You can see that this is the case with the file you provided:

$ file_info_dumper --event-list cmf_prediction_library_result_0000_ThreeFlavor_0_28000_0_00230.root                                                                                                 
==============================
File: cmf_prediction_library_result_0000_ThreeFlavor_0_28000_0_00230.root

Printing the list of Runs, SubRuns, and Events stored in the root file.

            Run         SubRun          Event
              1                              
              1              0               
              1              0              1

Even if downstream processes using RootInput do not retrieve event information or even drop events from the file, the framework still knows what events were processed in the initial job. This behavior is predicated on the assumption that (1) the framework is intended for processing events, and that (2) any processing of an initial output file does not fundamentally alter which events were processed when creating it.

The complication you are running into is that the framework sees one event in the file, and then it creates run, subrun, and event objects for processing that event. When creating the run, it sees two run fragments originating from two input files, attempts to merge the fragments and encounters an overlap in the event numbers, which is an error.

Even if you were to drop the events from the file you provided, the metadata in the file still records that event number (1, 0, 1) was processed, and combining that file with any other file processed with the same event number will yield an error.

To workaround this issue, you must ensure that the initial set of output files were produced with non-overlapping event numbers. This may seem like a limitation of the framework as you're not really interested in processing events--but in normal event-processing operations, this would not be an issue....provided you have been careful about event numbering.

What about relaxing art's assumptions?

This really cannot be done without significant analysis and effort. Paul Russo did in fact provide a mechanism for disabling art's event-number checking. But that was done for the art 2.13 release, which was never merged into the mainstream development branch. Such effort could be considered if the NOvA experiment deems it of high enough priority to make such a request.

What about adding a dataset field to the run-subrun-event number triplet?

I find this to be the most attractive solution, and we have wanted to go this route for sometime--but lack of effort prevents us. Again, if the NOvA experiment finds such a feature of high enough priority, then it should make the request.

Let me know if you wish to consider the discussion or whether I should just close this issue.

#9 Updated by Brian Rebel about 1 month ago

Hi Kyle

thanks for the explanation. How do I ensure the output files were produced with non-overlapping event numbers?

Should I be using the EmptyEvent source module to set the starting run number and simply not write anything into the run, subrun, or event record during the job? For example, add

source:
{
  module_type: EmptyEvent
  maxEvents:   10000       # Number of events to create
  firstRun:    1000001     # Run number to use for this file
  firstSubRun: 1           # Subrun number to use for this file
  firstEvent:  1           # number of first event in the file
}

to my job configuration?

#10 Updated by Kyle Knoepfel about 1 month ago

Brian, yes I think your proposal would be the best way to handle the issue. If you don't actually need to process events, I'd specify maxEvents as 1. I'd test this with a couple files before doing all 2499. :)

Let me know if you need any more direction.

#11 Updated by Kyle Knoepfel 15 days ago

Brian, can we close this issue?

#12 Updated by Brian Rebel 15 days ago

Please do.

#13 Updated by Kyle Knoepfel 15 days ago

  • % Done changed from 0 to 100
  • Assignee set to Kyle Knoepfel
  • Status changed from Feedback to Closed


Also available in: Atom PDF