Feature #22456

Provide experiment-specific hooks for customizing product instance names for fragments

Added by Kurt Biery almost 2 years ago. Updated about 2 months ago.

Target version:
Start date:
Due date:
% Done:


Estimated time:


DUNE folks are thinking about how to handle large data fragments, both at protoDUNE and thinking ahead to full DUNE. We have already done some testing that demonstrate that putting fragments of manageable size into art/ROOT data products with different product instance names helps.

Eric and I talked briefly about how to provide the ability for experiments to customize their product instance names, and he suggested extending the template concept that is already being used in SharedMemoryReader for specifying the mapping from fragment type (integer) to baseline product instance name (string). [SharedMemoryReader is used as the base class for the art InputSource in the EventBuilders.]

I have developed an initial implementation for this based on that model. I'll commit the candidate code to a branch in a few minutes and provide some additional information.


#1 Updated by Kurt Biery almost 2 years ago

The changes to SharedMemoryReader have been committed to the feature/22456_SMR_CustomProductInstanceNames branch in the artdaq repository. I have tested these changes in an artdaq-demo system on mu2edaq01, but I have not yet developed a sample experiment-specific class to fully test out the templating functionality. I'll try to do that for protoDUNE soon.

One side note: I had noticed that we often get a data product in the raw data files with a product instance name of "Container" and zero elements in that data product. I believe that I was able to simplify the determination of product instance names so that we don't get these empty data products. (I verified that this was true in my tests so far.)

#2 Updated by Eric Flumerfelt almost 2 years ago

I've taken a quick look at this change, and I think that we could probably get rid of the first template parameter to SharedMemoryReader:

class FragmentTypeToProductNameTranslator {
   virtual std::set<std::string> GetProductInstanceNames() = 0;
   virtual std::map<Fragment::type_t, std::string> GetFragmentTypeMap() = 0;
   virtual std::string GetInstanceNameForFragment(artdaq::detail::RawFragmentHeader const& fragment) = 0;
class DefaultFragmentTypeTranslator {
   virtual std::set<std::string> GetProductInstanceNames() {
      std::set<std::string> output; 
      auto map = GetFragmentTypeMap();
      for(auto& it : map) 
         if(!output.count(it->second)) output.insert(it->second);
      return output;

   virtual std::map<Fragment::type_t, std::string> GetFragmentTypeMap() { return artdaq::Fragment::MakeSystemTypeMap(); }

   virtual std::string GetInstanceNameForFragment(artdaq::Fragment const& fragment) {
     auto map = GetFragmentTypeMap();
     if(fragment.type() == artdaq::Fragment::ContainerFragmentType) {
        artdaq::ContainerFragment cf(fragment);
        if(map.count(cf.fragment_type()) return "Container" + map[cf.fragment_type()];
        return "Container";
     return map.count(fragment.type()) ? map[fragment.type()] : "unidentified" 

template<FragmentTypeToProductNameTranslator translator = artdaq::detail::DefaultFragmentTypeTranslator()>
struct SharedMemoryReader
...  Some changes to consider fragment-by-fragment (for splitting same Fragment type to multiple product instance names)

#3 Updated by Kurt Biery over 1 year ago

I've incorporated several of the suggested changes, such as moving the handling of the "Container" prefix into the Translator class. The changes have been committed to the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch. (Unfortunately, the changes on the develop branch associated with artdaq v3_05_00 were merged into the feature/22456_SMR_CustomProductInstanceNames branch, and I don't want to invest the time to resolve the merge conflicts with this latest round of SharedMemoryReader changes right now, hence the new branch.)

I have not yet removed the first template parameter, however. It seems like this would be a breaking change, and I don't want to break things in this way at this time.

#4 Updated by Kurt Biery over 1 year ago

To provide some more information on the status of the code changes for this Issue...

  • the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch was effectively created from the caabac6 commit hash in the artdaq repo.
  • the feature/22456_SMR_CustomProductInstanceNames (without the v1.5 suffix) is effectively obsolete at this point in time
  • the changes from the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch have been merged into the for_dune-artdaq branch
  • I don't believe that the changes from the feature/22456_SMR_CustomProductInstanceNames_v1.5 branch have been merged into the develop branch yet, because they have not yet been validated.

An item on my to-do list is to create a DemoInput<SomeDescriptiveSuffix> Input Source in the artdaq-demo repo to demonstrate the use of experiment-specific product instance names. The goal of that sample Input Source would be to help with the validation of this Issue.

#5 Updated by Eric Flumerfelt over 1 year ago

I have made a few changes and put them on a new branch, artdaq:feature/22456_CustomProductInstanceNames_v1.6. I have also created a demo input source which uses a custom translator on artdaq-demo:feature/22456_NthFragmentIDInput and tested these two branches with the mediumsystem_with_routing_master configuration.

#6 Updated by Eric Flumerfelt over 1 year ago

The v1.5 branch for this Issue has been merged into artdaq-v3_06_00. The v1.6 branch still needs validation.

#7 Updated by Kurt Biery about 1 year ago

I merged origin/develop into both artdaq:feature/22456_SMR_CustomProductInstanceNames_v1.6 and artdaq-demo:feature/22456_NthFragmentIDInput, for testing.

In the artdaq-demo repo, I needed to add a couple of "services" libraries to ArtModules/CMakeLists.txt to support the addition of the ArtdaqSharedMemoryServices to NthFragmentIDInput.

A test with the artdaq-demo using mediumsystem_with_routing_master looked fine.

However, I noticed that ArtModules/ and ArtModules/ do not have initialization of the fragment_type_labels_ data member, whereas
and do have that initialization. Is that intentional? If so, why?

#8 Updated by Eric Flumerfelt about 2 months ago

  • Status changed from Assigned to Closed

Also available in: Atom PDF