Project

General

Profile

Product Mixing

[ The product mixing feature was significantly extended for art 2.06.00, and the current version of this page reflects those improvements. For the earlier documentation, please see Product_Mixing@12. ]

Overview.

Product Mixing is our label for the ability to take a feed from a source of secondary events, and combine the products (usually collections) from multiple events into a single (possibly different type of) product or collection to be placed in the primary event. The motivation is of course obvious: background (e.g., cosmic) events or pile-up. Since art 2.06.00, this functionality has been extended to subrun and run products, although with certain caveats which are discussed later.

This feature is implemented by providing a module template, MixFilter as a skeleton to handle the low-level interaction with the primary event and secondary files, including the collection of multiple products from different secondary events and the registration of outgoing products. The user must provide a "detail" class implementation which has some mandatory and some optional members in order to register necessary products at the right time and mix the prepared groups of incoming products. The MixFilter will take care of reading the secondary data, either sequentially or randomly and cycling through multiple secondary input files as necessary.

In order to make the mixing task easier, we have provided several auxiliary utilities which will be described later.

In summary, the MixFilter module will do the following tasks:

  1. Construct a MixHelper object for use by the user's detail class, hereafter referred to as Detail (class) or detail (object).
  2. Construct detail, passing the helper and the module's parameter set. At this point, detail should register any mix operations (one per product to be mixed) using MixHelper::declareMixOp<>(), and any non-mixing products to go in to the event (e.g., bookkeeping objects) using MixHelper::produces<>().
  3. For each primary event:
    1. Call the optional Detail::startEvent() function if it exists so detail can position itself for the coming event.
    2. Call Detail::nSecondaries() to decide how many secondary events to read.
    3. Decide which secondary events are to be read.
    4. Call the optional Detail::processEventIDs() function in case the Detail class is interested in the sequence of event IDs (e.g., for bookkeeping purposes). The analogous optional Detail::processEventAuxiliaries() function will also be called at this time.
    5. Call the registered mixing operation for each product to be mixed in the order in which they were declared by detail, providing the secondary data to be mixed and putting the resultant mixed product into the primary event.
    6. Call the optional Detail::finalizeEvent() function to tidy up and possibly put any bookkeeping products (registered with MixHelper::produces<>()) into the primary event.

The MixFilter module will take care of opening new secondary files as necessary, and will additionally call an optional function Detail::eventsToSkip() as each file is opened (in sequential mode only) if it is desired to have a particular offset into each file to insure against coherent backgrounds in events.

Mix operations: what are they and what do they need to do?

A mix operation is a function provided by the user which returns a bool and expects three arguments:

  • vector<PROD const*> const& containing the secondary products to be mixed;
  • OPROD&, the output product to be filled by the mix function; and
  • PtrRemapper const&, a helper for use remapping Ptr and friends (see below).

It may be a free function, a member function of Detail or some other user class, or a function object ("functor") whose operator() has the correct signature (see source:art/Framework/IO/ProductMix/MixTypes.h for a definition of a useful template alias, art::MixFunc). This mix function supposed to use the information contained in the secondary products to fill the output product, using the PtrRemapper if appropriate, and return a bool indicating whether the output product should be placed in the event.

All mix operations must be declared at construction time by detail, using the MixHelper provided as a constructor argument.

Mix operations are always called in the event loop. The art::MixFilter module template and associated utilities take care of registering the primary output product, marshaling the secondary data streams, and obtaining and organizing the products from the secondary data stream for mixing into each primary event. If the mix function returns true, the output product will be written into the primary event.

For non-event-level products, the mix operation is called for every primary event with the sequence of subrun or run products corresponding to the events chosen for mixing from the secondary stream. The user is responsible for handling duplicates. If data are intended for the primary event, then the provided OPROD is the appropriate place; if data are intended for the primary subrun or run, then the detail object's endSubRun() or endRun() function(s) are responsible for putting those (explicitly declared) products into the primary data stream.

Note that since art::Ptr may not be used in subrun- and run-level products currently, the art::PtrRemapper provided to non-event-level mix operations is unintialized and should not be used.

Mandatory interface for Detail.

Constructor.

Detail(fhicl::ParameterSet const& ps, art::MixHelper& helper);

The constructor should use the provided parameter set to extract any configuration; and the helper to declare mix operations and any bookkeeping products with declareMixOp<>() and produces<>();

nSecondaries().

size_t nSecondaries();

Tell MixFilter how many secondaries to read for merging into this primary. If required, a random number engine may be obtained from the RandomNumberGenerator service.

Optional interface for Detail.

startEvent().

void startEvent(art::Event const& evt);

Do any per-event initialization of detail here.

eventsToSkip().

size_t eventsToSkip();

If provided and the MixFilter is in sequential secondary mode, this function will be called at the beginning of every file to ascertain how many events to skip before pulling products to merge.

processEventIDs().

void processEventIDs(art::EventIDSequence const& seq);

seq is an ordered sequence of art::EventID as defined in source:art/Framework/IO/ProductMix/MixTypes.h. Use as you will.

processEventAuxiliaries().

void processEventAuxiliaries(art::EventAuxiliarySequence const& seq);

seq is an ordered sequence of art::EventAuxiliary as defined in source:art/Framework/IO/ProductMix/MixTypes.h. Use as you will.

finalizeEvent().

void finalizeEvent(art::Event& evt);

Write any non-merge products (i.e. those you declared with MixHelper::produces<>()) to the event with evt.put<>().

respondToXXXX().

void respondToOpenInputFile(art::FileBlock const& fb);
void respondToCloseInputFile(art::FileBlock const& fb);
void respondToOpenOutputFiles(art::FileBlock const& fb);
void respondToCloseOutputFiles(art::FileBlock const& fb);

Equivalent to the eponymous functions of a module, called at the appropriate times.

beginSubRun()

void beginSubRun(art::SubRun const& sr);

Analogous to the eponymous module function, providing read-only access to the primary subrun.

endSubRun()

void endSubRun(art::SubRun& sr);

Analogous to the eponymous module function, providing write access to the primary subrun. Products registered for the primary subrun may be put() into the art::SubRun at this time.

beginRun()

void beginRun(art::Run const& r);

Analogous to the eponymous module function, providing read-only access to the primary run.

endRun()

void endRun(art::Run& r);

Analogous to the eponymous module function, providing write access to the primary run. Products registered for the primary run may be put() into the art::Run at this time.

Helper utilities.

MixHelper.

produces<>().

Analogous to producer or filter produces<>() calls: declare products to be put into the event, subrun or run which are not direct products of a single mix operation.

template <class PROD>
void produces(std::string const& instanceName = {});

template <class PROD, BranchType B>
void produces(std::string const& instanceName = {});

DeclareMixOp<>().

These function templates should be used to declare a mix operation.

All of the available declareMixOp() function templates have the following template arguments:

  1. The BranchType (defaults to art::InEvent). Specify explicitly if you wish to mix subrun or run products, or if you need to specify explicitly any of the other template arguments.
  2. The incoming product type (deduced from the provided callable mixer argument).
  3. The outgoing product type (deduced from the provided callable mixer argument).

Mix operations operation may be specified by providing to the appropriate declareMixOp<>() function:

  1. an InputTag specifying which secondary products should be mixed;
  2. an optional instance label for the mixed product (defaulting to the instance label of the incoming product if unspecified); and
  3. a callable mixer such as:
bool mixfunc(std::vector<PROD const*> const&,
             OPROD&,
             PtrRemapper const&);
As the user may prefer, the mixer may take the form of:
  • an arbitrarily-named free function, or
  • a function object whose operator() must have the correct type (see mixfunc above), or
  • an arbitrarily-named member function of any class. In each case, the mixer must have the same type (i.e., the same return type and the same parameter types) illustrated by mixfunc above. The return value of the mix function is taken to indicate whether the product should be placed in the event.
    For most situations, the product type template arguments need not be specified as they can be deduced from the signature of the provided callable object.
    If the provided callable object is a member function it may be provided bound to the object upon which it is to be called by the user (in which case it is treated as a free function by the registration method) or by specifying the member function followed by the object to which it should be bound (in which case the bind will be done for the user). In this latter case the template argument specifying the product types need not be specified usually as they may be deduced from the signature of the provided function. If one specifies an overload set however (e.g., in the case where a class has several mix() member functions, each with arguments indicating a different product) then either the template arguments must be specified in order to constrain the overload set to a single function, or the function argument must be const_cast<>() to specify the desired signature.
  1. An optional bool, "outputProduct," defaulting to, "true." A false value for this parameter indicates that the mix product will never be put into the event and should therefore not be declared by the MixHelper. If the mix operation so registered ever returns true an exception will be thrown.

declareMixOp<>() may be called with any of the following argument combinations:

  1. Provide an InputTag and a mixer that is a free function or function object (or pre-bound member function).
  2. Provide an InputTag, an output instance label, and a mixer that is a free function or function object (or pre-bound member function).
  3. Provide an InputTag, a mixer that is a non-const member function (of any class), and an object to which that member function should be bound.
  4. Provide an InputTag, an output instance label, a mixer that is a non-const member function (of any class), and an object to which that member function should be bound.
  5. As 3, but with a const member function.
  6. As 4, but with a const member function.

CollectionUtilities.h: concatContainers() and flattenCollections().

This header provides two utilities for dealing with the merging of collections:

concatContainers().

template <class CONTAINER>
void
concatContainers(CONTAINER& out, CONTAINER const& in);

Append the contents of container in to those of container out.

flattenCollections().

template <class COLLECTION>
void
flattenCollections(vector<COLLECTION const*> in, COLLECTION& out);

template <class COLLECTION, class OFFSETS>
void
flattenCollections(vector<COLLECTION const*> in,
                   COLLECTION& out,
                   OFFSETS& offsets);

Given a sequence of collections by const* accumulate them into out, possibly storing the offsets (for use in remapping Ptr into these collections) for use by PtrRemapper (see below). N.B. if COLLECTION is a PtrVector<T> then the extra check is made that the individual PtrVector<T> are compatible (ie point to the same product). If this is not the case, you should merge their underlying collections into a single merged product first (in a separate merge operation) and then use PtrRemapper to re-base and merge the PtrVector<T> collections.

PtrRemapper.

If at some point you have one or more Ptr objects pointing to collections which have been merged, they will need to be re-based against the merged collection. This will mean changing the underlying ProductID of the pointer-to product and also adjusting the offset by an amount related to the pointed-to collection's position within the new, merged collection. It goes without saying therefore, that the underlying collections should be merged before attempting to re-base any Ptr into them.

Each merge function, every event, is provided via its interface with a reference to a PtrRemapper functor which is primed to be capable of re-basing Ptr objects in the current event. PtrRemapper has several operator() templates that accomplish this. Given PtrRemapper const& remamp:

1. Remap a single Ptr.

Ptr<A> newPtr(remap(oldPtr, offset));

2. Remap a single PtrVector.
PtrVector<A> newPV(remap(oldPV, offset));

3. Remap a compatible collection (including PtrVector) of Ptr providing begin, end iterators (This will also remap a compatible collection of PtrVector, but not of PtrVector const* -- for the latter, see 4-10.)

PtrVector<A> newPV;
remap(oldPV.begin(),
      oldPV.end(),
      std::back_inserter(newPV),
      offset);

4. Remap and flatten a set of products which are containers of Ptrs (which includes PtrVector).

remap(in, out, offsets);
where offsets is likely calculated by the appropriate call to art::flattenCollections<>(). See source:art/Persistency/Common/CollectionUtilities.h for details.
5. Remap and flatten a set of containers of Ptrs (including PtrVector) which may be obtained from a component of the provided product. Provide a free function of the correct signature to return a reference to the container of Ptrs given a secondary product, e.g.:

PtrVector<B> const& myfunc(A const* prod)
{
  return prod->myBs();
}

remap(in, out, offsets, &myfunc);

6. Remap and flatten a set of containers of Ptrs (including PtrVector) which may be obtained from a component of the provided product. Provide the name of a member function of the provided product which is an accessor for the container (taking no arguments).

remap(in, out, offsets, &A::myBs);

7. Remap and flatten a set of containers of Ptrs (including PtrVector) which may be obtained from a component of the provided product. Provide the name of a member datum of the provided product which is the container.
remap(in, out, offsets, &A::myBs);

8. Remap and flatten a set of containers of Ptrs (including PtrVector) which is a component of the provided product using the provided accessor member function of a class which is not the product.

class Aprocessor {
public:
  B const& myBs(A const*);
};

Aprocessor myAp;

remap(in, out, offsets, Aprocessor::myBs, myAp);

Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:

const_cast<Aprocessor&>(myAp);

9. Remap and flatten a set of containers of Ptrs (including PtrVector) which is a component of the provided product using the provided const accessor member function of a class which is not the product.

class Aprocessor {
public:
  B const& myBs(A const*) const;
};

Aprocessor myAp;

remap(in, out, offsets, Aprocessor::myBs, myAp);

Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:

const_cast<Aprocessor const&>(myAp);

10. More general version of 5-9 that takes a final argument which is of arbitrary type provided it or its operator() has the correct signature. The drawback is that one of the template arguments (CONT, specifying the type of the collection of Ptrs you wish to remap) is not deducible, meaning that instead of:

remap(...);

one must type, for example:

remap.operator()<std::vector<art::Ptr<B>>>(...)

Therefore, 4-9 are the recommended signatures for straightforward client code -- this one is provided for maximum flexibility and for use internally by signatures 5-9.

Examples.