Product Mixing » History » Version 5

« Previous - Version 5/19 (diff) - Next » - Current version
Christopher Green, 10/27/2011 07:00 AM

Product Mixing


Product Mixing is our label for the ability to take a feed from a source of secondary events, and combine the products (usually collections) from multiple events into a single product or collection to be placed in the primary event. The motivation is of course obvious: background (eg cosmic) events or pile-up.

This feature is implemented by providing a module template, MixFilter as a skeleton to handle the low-level interaction with the primary event and secondary files, including the collection of multiple products from different secondary events and the registration of outgoing products. The user must provide a "detail" class implementation which has some mandatory and some optional members in order to register necessary products at the right time and mix the prepared groups of incoming products. The MixFilter will take care of reading the secondary data, either sequentially or randomly and cycling through multiple secondary input files as necessary.

In order to make the mixing task easier, we have provided several auxiliary utilities which will be described later.

In summary, the MixFilter module will do the following tasks:

  1. Construct a MixHelper object for use by the user's detail class, hereafter referred to as Detail (class) or detail (object).
  2. Construct detail, passing the helper and the module's parameter set. At this point, detail should register any mix operations (one per product to be mixed) using MixHelper::declareMixOp<>(), and any non-mixing products to go in to the event (e.g. bookkeeping objects) using MixHelper::produces<>().
  3. For each primary event:
    1. Call the optional Detail::startEvent() function if it exists so detail can position itself for the coming event.
    2. Call Detail::nSecondaries() to decide how many secondary events to read.
    3. Decide which secondary events are to be read.
    4. Call the optional Detail::processEventIDs() function in case the Detail class is interested in the sequence of event IDs (e.g. for bookkeeping purposes).
    5. Call the registered mixing operation for each product to be mixed, providing the secondary data to be mixed and putting the resultant mixed product into the primary event.
    6. Call the optional Detail::finalizeEvent() function to tidy up and possibly put any bookkeeping products (registered with MixHelper::produces<>()) into the primary event.

The MixFilter module will take care of opening new secondary files as necessary, and will additionally call an optional function Detail::eventsToSkip() as each file is opened (in sequential mode only) if it is desired to have a particular offset into each file to insure against coherent backgrounds in events.

Mix operations: what are they and what do they need to do?

A mix operation is a function provided by the user which returns a bool and expects three arguments:

  • vector<PROD const *> const & containing the secondary products to be mixed;
  • PROD &, the output product to be filled by the mix function; and
  • PtrRemapper const &, a helper for use remapping Ptr and friends (see below).

It may be a free function, a member function of Detail or some other user class, or a function object ("functor") whose operator() has the correct signature. It is supposed to use the information contained in the secondary products to fill the output product, using the PtrRemapper if necessary, and return a bool indicating whether the output product should be placed in the event.

All mix operations must be declared at construction time by detail, using the MixHelper provided as a constructor argument.

Mandatory interface for Detail.


Detail(fhicl::ParameterSet const & ps, art::MixHelper & helper);
The constructor should use the provided parameter set to extract any configuration; and the helper to declare mix operations and any bookkeeping products with declareMixOp<>() and @produces<>();


size_t nSecondaries();
Tell MixFilter how many secondaries to read for merging into this primary. If required, a random number engine may be obtained from the RandomNumberGenerator service.

Optional interface for Detail.


void startEvent(art::Event const & evt);
Do any per-event initialization of detail here.


size_t eventsToSkip();
If provided and the MixFilter is in sequential secondary mode, this function will be called at the beginning of every file to ascertain how many events to skip before pulling products to merge.


void processEventIDs(art::EventIDSequence const & seq);
seq is an ordered sequence of art::EventID. Use as you will.


void finalizeEvent(art::Event & evt);
Write any non-merge products (i.e. those you declared with MixHelper::produces<>()) to the event with evt.put<>().

Helper utilities.



Analogous to producer or filter produces<>() calls: declare products to be put into the event which are not mix products.

template <class PROD>
void produces(std::string const & instanceName = std::string());

template <class PROD, BranchType B>
void produces(std::string const & instanceName = std::string());


These function templates should be used to declare a mix operation.

Such an operation may be specified by providing:

  1. an InputTag specifying which secondary products should be mixed;
  2. an optional instance label for the mixed product (defaulting to the instance label of the incoming product if unspecified); and
  3. a callable mixer such as:
    bool mixfunc(std::vector<PROD const *> const &,
                  PROD &,
                  PtrRemapper const &);
    As the user may prefer, the mixer may take the form of:
    • an arbitrarily-named free function, or
    • a function object whose operator() must have the correct type (see mixfunc above), or
    • an arbitrarily-named member function of any class. In each case, the mixer must have the same type (i.e., the same return type and the same parameter types) illustrated by mixfunc above. The return value of the mix function is taken to indicate whether the product should be placed in the event.
      For free functions, function objects, and pre-bound member functions, the product type template argument need not be specified as it can be deduced from the signature of the provided function.
      If the provided function is a member function it may be provided bound to the object upon which it is to be called by the user (in which case it is treated as a free function by the registration method) or by specifying the member function followed by the object to which it should be bound (in which case the bind will be done for the user). In this latter case the template argument specifying the product type need not be specified usually as it may be deduced from the signature of the provided function. If one specifies an overload set however (e.g. in the case where a class has several mix() member functions, each with arguments indicating a different product) then the template argument must be specified in order to constrain the overload set to a single function.
  4. An optional bool, "outputProduct," defaulting to, "true." A false value for this parameter indicates that the mix product will never be put into the event and should therefore not be declared by the MixHelper. If the mix operation so registered ever returns true an exception will be thrown.

declareMixOp<>() may be called with any of the following argument combinations:

  1. Provide an InputTag and a mixer that is a free function or function object (or pre-bound member function).
  2. Provide an InputTag, an output instance label, and a mixer that is a free function or function object (or pre-bound member function).
  3. Provide an InputTag, a mixer that is a non-const member function (of any class), and an object to which that member function should be bound.
  4. Provide an InputTag, an output instance label, a mixer that is a non-const member function (of any class), and an object to which that member function should be bound.
  5. As 3, but with a const member function.
  6. As 4, but with a const member function.

CollectionUtilities.h: concatContainers() and flattenCollections().

This header provides two utilities for dealing with the merging of collections:


template <class CONTAINER>
concatContainers(CONTAINER & out, CONTAINER const & in);

Append the contents of container in to those of container out.


template <class COLLECTION>
flattenCollections(vector<COLLECTION const *> in, COLLECTION & out);

template <class COLLECTION, class OFFSETS>
flattenCollections(vector<COLLECTION const *> in,
                   COLLECTION & out,
                   OFFSETS & offsets);

Given a sequence of collections by const * accumulate them into out, possibly storing the offsets (for use in remapping Ptr into these collections) for use by PtrRemapper (see below). N.B. if COLLECTION is a PtrVector<T> then the extra check is made that the individual PtrVector<T> are compatible (ie point to the same product). If this is not the case, you should merge their underlying collections into a single merged product first (in a separate merge operation) and then use PtrRemapper to re-base and merge the PtrVector<T> collections.