Project

General

Profile

Product Mixing » History » Version 8

« Previous - Version 8/19 (diff) - Next » - Current version
Christopher Green, 10/27/2011 09:21 AM


Product Mixing

Overview.

Product Mixing is our label for the ability to take a feed from a source of secondary events, and combine the products (usually collections) from multiple events into a single product or collection to be placed in the primary event. The motivation is of course obvious: background (eg cosmic) events or pile-up.

This feature is implemented by providing a module template, MixFilter as a skeleton to handle the low-level interaction with the primary event and secondary files, including the collection of multiple products from different secondary events and the registration of outgoing products. The user must provide a "detail" class implementation which has some mandatory and some optional members in order to register necessary products at the right time and mix the prepared groups of incoming products. The MixFilter will take care of reading the secondary data, either sequentially or randomly and cycling through multiple secondary input files as necessary.

In order to make the mixing task easier, we have provided several auxiliary utilities which will be described later.

In summary, the MixFilter module will do the following tasks:

  1. Construct a MixHelper object for use by the user's detail class, hereafter referred to as Detail (class) or detail (object).
  2. Construct detail, passing the helper and the module's parameter set. At this point, detail should register any mix operations (one per product to be mixed) using MixHelper::declareMixOp<>(), and any non-mixing products to go in to the event (e.g. bookkeeping objects) using MixHelper::produces<>().
  3. For each primary event:
    1. Call the optional Detail::startEvent() function if it exists so detail can position itself for the coming event.
    2. Call Detail::nSecondaries() to decide how many secondary events to read.
    3. Decide which secondary events are to be read.
    4. Call the optional Detail::processEventIDs() function in case the Detail class is interested in the sequence of event IDs (e.g. for bookkeeping purposes).
    5. Call the registered mixing operation for each product to be mixed, providing the secondary data to be mixed and putting the resultant mixed product into the primary event.
    6. Call the optional Detail::finalizeEvent() function to tidy up and possibly put any bookkeeping products (registered with MixHelper::produces<>()) into the primary event.

The MixFilter module will take care of opening new secondary files as necessary, and will additionally call an optional function Detail::eventsToSkip() as each file is opened (in sequential mode only) if it is desired to have a particular offset into each file to insure against coherent backgrounds in events.

Mix operations: what are they and what do they need to do?

A mix operation is a function provided by the user which returns a bool and expects three arguments:

  • vector<PROD const *> const & containing the secondary products to be mixed;
  • PROD &, the output product to be filled by the mix function; and
  • PtrRemapper const &, a helper for use remapping Ptr and friends (see below).

It may be a free function, a member function of Detail or some other user class, or a function object ("functor") whose operator() has the correct signature. It is supposed to use the information contained in the secondary products to fill the output product, using the PtrRemapper if necessary, and return a bool indicating whether the output product should be placed in the event.

All mix operations must be declared at construction time by detail, using the MixHelper provided as a constructor argument.

Mandatory interface for Detail.

Constructor.

Detail(fhicl::ParameterSet const & ps, art::MixHelper & helper);
The constructor should use the provided parameter set to extract any configuration; and the helper to declare mix operations and any bookkeeping products with declareMixOp<>() and @produces<>();

nSecondaries().

size_t nSecondaries();
Tell MixFilter how many secondaries to read for merging into this primary. If required, a random number engine may be obtained from the RandomNumberGenerator service.

Optional interface for Detail.

startEvent().

void startEvent(art::Event const & evt);
Do any per-event initialization of detail here.

eventsToSkip().

size_t eventsToSkip();
If provided and the MixFilter is in sequential secondary mode, this function will be called at the beginning of every file to ascertain how many events to skip before pulling products to merge.

processEventIDs().

void processEventIDs(art::EventIDSequence const & seq);
seq is an ordered sequence of art::EventID. Use as you will.

finalizeEvent().

void finalizeEvent(art::Event & evt);
Write any non-merge products (i.e. those you declared with MixHelper::produces<>()) to the event with evt.put<>().

Helper utilities.

MixHelper.

produces<>().

Analogous to producer or filter produces<>() calls: declare products to be put into the event which are not mix products.

template <class PROD>
void produces(std::string const & instanceName = std::string());

template <class PROD, BranchType B>
void produces(std::string const & instanceName = std::string());

DeclareMixOp<>().

These function templates should be used to declare a mix operation.

Such an operation may be specified by providing:

  1. an InputTag specifying which secondary products should be mixed;
  2. an optional instance label for the mixed product (defaulting to the instance label of the incoming product if unspecified); and
  3. a callable mixer such as:
    bool mixfunc(std::vector<PROD const *> const &,
                 PROD &,
                 PtrRemapper const &);
    As the user may prefer, the mixer may take the form of:
    • an arbitrarily-named free function, or
    • a function object whose operator() must have the correct type (see mixfunc above), or
    • an arbitrarily-named member function of any class. In each case, the mixer must have the same type (i.e., the same return type and the same parameter types) illustrated by mixfunc above. The return value of the mix function is taken to indicate whether the product should be placed in the event.
      For free functions, function objects, and pre-bound member functions, the product type template argument need not be specified as it can be deduced from the signature of the provided function.
      If the provided function is a member function it may be provided bound to the object upon which it is to be called by the user (in which case it is treated as a free function by the registration method) or by specifying the member function followed by the object to which it should be bound (in which case the bind will be done for the user). In this latter case the template argument specifying the product type need not be specified usually as it may be deduced from the signature of the provided function. If one specifies an overload set however (e.g. in the case where a class has several mix() member functions, each with arguments indicating a different product) then the template argument must be specified in order to constrain the overload set to a single function.
  4. An optional bool, "outputProduct," defaulting to, "true." A false value for this parameter indicates that the mix product will never be put into the event and should therefore not be declared by the MixHelper. If the mix operation so registered ever returns true an exception will be thrown.

declareMixOp<>() may be called with any of the following argument combinations:

  1. Provide an InputTag and a mixer that is a free function or function object (or pre-bound member function).
  2. Provide an InputTag, an output instance label, and a mixer that is a free function or function object (or pre-bound member function).
  3. Provide an InputTag, a mixer that is a non-const member function (of any class), and an object to which that member function should be bound.
  4. Provide an InputTag, an output instance label, a mixer that is a non-const member function (of any class), and an object to which that member function should be bound.
  5. As 3, but with a const member function.
  6. As 4, but with a const member function.

CollectionUtilities.h: concatContainers() and flattenCollections().

This header provides two utilities for dealing with the merging of collections:

concatContainers().

template <class CONTAINER>
void
concatContainers(CONTAINER & out, CONTAINER const & in);

Append the contents of container in to those of container out.

flattenCollections().

template <class COLLECTION>
void
flattenCollections(vector<COLLECTION const *> in, COLLECTION & out);

template <class COLLECTION, class OFFSETS>
void
flattenCollections(vector<COLLECTION const *> in,
                   COLLECTION & out,
                   OFFSETS & offsets);

Given a sequence of collections by const * accumulate them into out, possibly storing the offsets (for use in remapping Ptr into these collections) for use by PtrRemapper (see below). N.B. if COLLECTION is a PtrVector<T> then the extra check is made that the individual PtrVector<T> are compatible (ie point to the same product). If this is not the case, you should merge their underlying collections into a single merged product first (in a separate merge operation) and then use PtrRemapper to re-base and merge the PtrVector<T> collections.

PtrRemapper.

If at some point you have one or more Ptr objects pointing to collections which have been merged, they will need to be re-based against the merged collection. This will mean changing the underlying ProductID of the pointer-to product and also adjusting the offset by an amount related to the pointed-to collection's position within the new, merged collection. It goes without saying therefore, that the underlying collections should be merged before attempting to re-base any Ptr into them.

Each merge function, every event, is provided via its interface with a reference to a PtrRemapper functor which is primed to be capable of re-basing Ptr objects in the current event. PtrRemapper has several operator() templates that accomplish this. Given PtrRemapper const & remamp:

  1. Remap a single Ptr.
    Ptr<A> newPtr(remap(oldPtr, offset));
  2. Remap a single PtrVector.
    PtrVector<A> newPV(remap(oldPV, offset));
  3. Remap a compatible collection (including PtrVector) of Ptr providing begin, end iterators (This will also remap a compatible collection of PtrVector, but not of PtrVector const * -- for the latter, see 4-10.)
    PtrVector<A> newPV;
          remap(oldPV.begin(),
                oldPV.end(),
                std::back_inserter(newPV),
                offset);
  4. Remap and flatten a set of products which are containers of Ptrs (which includes PtrVector).
    remap(in, out, offsets);
    where offsets is likely calculated by the appropriate call to art::flattenCollections<>(). See source:art/Persistency/Common/CollectionUtilities.h for details.
  5. Remap and flatten a set of containers of Ptrs (including PtrVector) which may be obtained from a component of the provided product. Provide a free function of the correct signature to return a reference to the container of Ptrs given a secondary product, e.g.:
    PtrVector<B> const &myfunc(A const *prod) {
      return prod->myBs();
    }
    
    remap(in, out, offsets, &myfunc);
  6. Remap and flatten a set of containers of Ptrs (including PtrVector) which may be obtained from a component of the provided product. Provide the name of a member function of the provided product which is an accessor for the container (taking no arguments).
    remap(in, out, offsets, &A::myBs);
  7. Remap and flatten a set of containers of Ptrs (including PtrVector) which may be obtained from a component of the provided product. Provide the name of a member datum of the provided product which is the container.
    remap(in, out, offsets, &A::myBs);
  8. Remap and flatten a set of containers of Ptrs (including PtrVector) which is a component of the provided product using the provided accessor member function of a class which is not the product.
    class Aprocessor {
    public:
      B const &myBs(A const *);
    };
    
    Aprocessor myAp;
    
    remap(in, out, offsets, Aprocessor::myBs, myAp);
    Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:
    const_cast<Aprocessor &>(myAp);
  9. Remap and flatten a set of containers of Ptrs (including PtrVector) which is a component of the provided product using the provided const accessor member function of a class which is not the product.
    class Aprocessor {
    public:
      B const &myBs(A const *) const;
    };
    
    Aprocessor myAp;
    
    remap(in, out, offsets, Aprocessor::myBs, myAp);
    Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:
    const_cast<Aprocessor const &>(myAp);
  10. More general version of 5-9 that takes a final argument which is of arbitrary type provided it or its operator() has the correct signature. The drawback is that one of the template arguments (CONT, specifying the type of the collection of Ptrs you wish to remap) is not deducible, meaning that instead of:
    remap(...);
    one must type (e.g.):
    remap.operator()<std::vector<art::Ptr<B> > >(...)
    Therefore, 4-9 are the recommended signatures for straightforward client code -- this one is provided for maximum flexibility.

Examples.

source:test/Integration/MixFilterTest_module.cc

mu2e/Offline/EventMixing/src/MixMCEvents_module.cc