Feature #22456

Provide experiment-specific hooks for customizing product instance names for fragments

Added by Kurt Biery about 2 months ago. Updated 3 days ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:


DUNE folks are thinking about how to handle large data fragments, both at protoDUNE and thinking ahead to full DUNE. We have already done some testing that demonstrate that putting fragments of manageable size into art/ROOT data products with different product instance names helps.

Eric and I talked briefly about how to provide the ability for experiments to customize their product instance names, and he suggested extending the template concept that is already being used in SharedMemoryReader for specifying the mapping from fragment type (integer) to baseline product instance name (string). [SharedMemoryReader is used as the base class for the art InputSource in the EventBuilders.]

I have developed an initial implementation for this based on that model. I'll commit the candidate code to a branch in a few minutes and provide some additional information.


#1 Updated by Kurt Biery about 2 months ago

The changes to SharedMemoryReader have been committed to the feature/22456_SMR_CustomProductInstanceNames branch in the artdaq repository. I have tested these changes in an artdaq-demo system on mu2edaq01, but I have not yet developed a sample experiment-specific class to fully test out the templating functionality. I'll try to do that for protoDUNE soon.

One side note: I had noticed that we often get a data product in the raw data files with a product instance name of "Container" and zero elements in that data product. I believe that I was able to simplify the determination of product instance names so that we don't get these empty data products. (I verified that this was true in my tests so far.)

#2 Updated by Eric Flumerfelt about 2 months ago

I've taken a quick look at this change, and I think that we could probably get rid of the first template parameter to SharedMemoryReader:

class FragmentTypeToProductNameTranslator {
   virtual std::set<std::string> GetProductInstanceNames() = 0;
   virtual std::map<Fragment::type_t, std::string> GetFragmentTypeMap() = 0;
   virtual std::string GetInstanceNameForFragment(artdaq::detail::RawFragmentHeader const& fragment) = 0;
class DefaultFragmentTypeTranslator {
   virtual std::set<std::string> GetProductInstanceNames() {
      std::set<std::string> output; 
      auto map = GetFragmentTypeMap();
      for(auto& it : map) 
         if(!output.count(it->second)) output.insert(it->second);
      return output;

   virtual std::map<Fragment::type_t, std::string> GetFragmentTypeMap() { return artdaq::Fragment::MakeSystemTypeMap(); }

   virtual std::string GetInstanceNameForFragment(artdaq::Fragment const& fragment) {
     auto map = GetFragmentTypeMap();
     if(fragment.type() == artdaq::Fragment::ContainerFragmentType) {
        artdaq::ContainerFragment cf(fragment);
        if(map.count(cf.fragment_type()) return "Container" + map[cf.fragment_type()];
        return "Container";
     return map.count(fragment.type()) ? map[fragment.type()] : "unidentified" 

template<FragmentTypeToProductNameTranslator translator = artdaq::detail::DefaultFragmentTypeTranslator()>
struct SharedMemoryReader
...  Some changes to consider fragment-by-fragment (for splitting same Fragment type to multiple product instance names)

#3 Updated by Kurt Biery about 2 months ago

I've incorporated several of the suggested changes, such as moving the handling of the "Container" prefix into the Translator class. The changes have been committed to the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch. (Unfortunately, the changes on the develop branch associated with artdaq v3_05_00 were merged into the feature/22456_SMR_CustomProductInstanceNames branch, and I don't want to invest the time to resolve the merge conflicts with this latest round of SharedMemoryReader changes right now, hence the new branch.)

I have not yet removed the first template parameter, however. It seems like this would be a breaking change, and I don't want to break things in this way at this time.

#4 Updated by Kurt Biery about 1 month ago

To provide some more information on the status of the code changes for this Issue...

  • the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch was effectively created from the caabac6 commit hash in the artdaq repo.
  • the feature/22456_SMR_CustomProductInstanceNames (without the v1.5 suffix) is effectively obsolete at this point in time
  • the changes from the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch have been merged into the for_dune-artdaq branch
  • I don't believe that the changes from the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch have been merged into the develop branch yet, because they have not yet been validated.

An item on my to-do list is to create a DemoInput<SomeDescriptiveSuffix> Input Source in the artdaq-demo repo to demonstrate the use of experiment-specific product instance names. The goal of that sample Input Source would be to help with the validation of this Issue.

#5 Updated by Eric Flumerfelt 3 days ago

I have made a few changes and put them on a new branch, artdaq:feature/22456_CustomProductInstanceNames_v1.6. I have also created a demo input source which uses a custom translator on artdaq-demo:feature/22456_NthFragmentIDInput and tested these two branches with the mediumsystem_with_routing_master configuration.

Also available in: Atom PDF