Project

General

Profile

Product Mixing » History » Version 12

Christopher Green, 10/27/2011 04:54 PM

1 1 Christopher Green
h1. Product Mixing
2 1 Christopher Green
3 1 Christopher Green
h2. Overview.
4 1 Christopher Green
5 10 Walter E Brown
*Product Mixing* is our label for the ability to take a feed from a source of secondary events, and combine the products (usually collections) from multiple events into a single product or collection to be placed in the primary event. The motivation is of course obvious: background (e.g., cosmic) events or pile-up.
6 1 Christopher Green
7 1 Christopher Green
This feature is implemented by providing a module template, @MixFilter@ as a skeleton to handle the low-level interaction with the primary event and secondary files, including the collection of multiple products from different secondary events and the registration of outgoing products. The user must provide a "detail" class implementation which has some mandatory and some optional members in order to register necessary products at the right time and mix the prepared groups of incoming products. The @MixFilter@ will take care of reading the secondary data, either sequentially or randomly and cycling through multiple secondary input files as necessary.
8 1 Christopher Green
9 1 Christopher Green
In order to make the mixing task easier, we have provided several auxiliary utilities which will be described later.
10 1 Christopher Green
11 8 Christopher Green
In summary, the @MixFilter@ module will do the following tasks:
12 1 Christopher Green
13 1 Christopher Green
# Construct a @MixHelper@ object for use by the user's detail class, hereafter referred to as @Detail@ (class) or @detail@ (object).
14 10 Walter E Brown
# Construct @detail@, passing the helper and the module's parameter set. At this point, @detail@ should register any mix operations (one per product to be mixed) using @MixHelper::declareMixOp<>()@, and any non-mixing products to go in to the event (e.g., bookkeeping objects) using @MixHelper::produces<>()@.
15 1 Christopher Green
# For each primary event:
16 1 Christopher Green
## Call the optional @Detail::startEvent()@ function if it exists so @detail@ can position itself for the coming event.
17 1 Christopher Green
## Call @Detail::nSecondaries()@ to decide how many secondary events to read.
18 1 Christopher Green
## Decide which secondary events are to be read.
19 10 Walter E Brown
## Call the optional @Detail::processEventIDs()@ function in case the Detail class is interested in the sequence of event IDs (e.g., for bookkeeping purposes).
20 12 Christopher Green
## Call the registered mixing operation for each product to be mixed _in the order in which they were declared by_ @detail@, providing the secondary data to be mixed and putting the resultant mixed product into the primary event.
21 1 Christopher Green
## Call the optional @Detail::finalizeEvent()@ function to tidy up and possibly put any bookkeeping products (registered with @MixHelper::produces<>()@) into the primary event.
22 1 Christopher Green
23 1 Christopher Green
The @MixFilter@ module will take care of opening new secondary files as necessary, and will additionally call an optional function @Detail::eventsToSkip()@ as each file is opened (in sequential mode only) if it is desired to have a particular offset into each file to insure against coherent backgrounds in events.
24 1 Christopher Green
25 2 Christopher Green
h2. Mix operations: what are they and what do they need to do?
26 2 Christopher Green
27 3 Christopher Green
A mix operation is a function provided by the user which returns a @bool@ and expects three arguments:
28 3 Christopher Green
29 3 Christopher Green
* @vector<PROD const *> const &@ containing the secondary products to be mixed;
30 3 Christopher Green
* @PROD &@, the output product to be filled by the mix function; and
31 3 Christopher Green
* @PtrRemapper const &@, a helper for use remapping @Ptr@ and friends (see below).
32 3 Christopher Green
33 3 Christopher Green
It may be a free function, a member function of @Detail@ or some other user class, or a function object ("functor") whose @operator()@ has the correct signature. It is supposed to use the information contained in the secondary products to fill the output product, using the @PtrRemapper@ if necessary, and return a @bool@ indicating whether the output product should be placed in the event.
34 3 Christopher Green
35 3 Christopher Green
All mix operations must be declared _at construction time_ by @detail@, using the @MixHelper@ provided as a constructor argument.
36 3 Christopher Green
37 1 Christopher Green
h2. Mandatory interface for @Detail@.
38 1 Christopher Green
39 3 Christopher Green
h3. Constructor.
40 3 Christopher Green
41 9 Christopher Green
<pre><code class="cplusplus">Detail(fhicl::ParameterSet const & ps, art::MixHelper & helper);</code></pre>The constructor should use the provided parameter set to extract any configuration; and the helper to declare mix operations and any bookkeeping products with @declareMixOp<>()@ and @produces<>()@;
42 3 Christopher Green
43 3 Christopher Green
h3. @nSecondaries()@.
44 3 Christopher Green
45 4 Christopher Green
<pre><code class="cplusplus">size_t nSecondaries();</code></pre>Tell @MixFilter@ how many secondaries to read for merging into this primary. If required, a random number engine may be obtained from the @RandomNumberGenerator@ service.
46 3 Christopher Green
47 1 Christopher Green
h2. Optional interface for @Detail@.
48 3 Christopher Green
49 3 Christopher Green
h3. @startEvent()@.
50 1 Christopher Green
51 4 Christopher Green
<pre><code class="cplusplus">void startEvent(art::Event const & evt);</code></pre>Do any per-event initialization of @detail@ here.
52 1 Christopher Green
53 4 Christopher Green
54 1 Christopher Green
h3. @eventsToSkip()@.
55 3 Christopher Green
56 4 Christopher Green
<pre><code class="cplusplus">size_t eventsToSkip();</code></pre>If provided and the @MixFilter@ is in sequential secondary mode, this function will be called at the beginning of every file to ascertain how many events to skip before pulling products to merge.
57 3 Christopher Green
58 4 Christopher Green
59 3 Christopher Green
h3. @processEventIDs()@.
60 1 Christopher Green
61 4 Christopher Green
<pre><code class="cplusplus">void processEventIDs(art::EventIDSequence const & seq);</code></pre>@seq@ is an ordered sequence of @art::EventID@. Use as you will.
62 1 Christopher Green
63 4 Christopher Green
64 3 Christopher Green
h3. @finalizeEvent()@.
65 3 Christopher Green
66 4 Christopher Green
<pre><code class="cplusplus">void finalizeEvent(art::Event & evt);</code></pre>Write any non-merge products (i.e. those you declared with @MixHelper::produces<>()@) to the event with @evt.put<>()@.
67 1 Christopher Green
68 4 Christopher Green
69 1 Christopher Green
h2. Helper utilities.
70 1 Christopher Green
71 1 Christopher Green
h3. @MixHelper@.
72 4 Christopher Green
73 4 Christopher Green
h4. @produces<>()@.
74 4 Christopher Green
75 4 Christopher Green
Analogous to producer or filter @produces<>()@ calls: declare products to be put into the event which are _not_ mix products.<pre><code class="cplusplus">template <class PROD>
76 4 Christopher Green
void produces(std::string const & instanceName = std::string());
77 4 Christopher Green
78 4 Christopher Green
template <class PROD, BranchType B>
79 4 Christopher Green
void produces(std::string const & instanceName = std::string());</code></pre>
80 4 Christopher Green
81 4 Christopher Green
h4. @DeclareMixOp<>()@.
82 4 Christopher Green
83 4 Christopher Green
These function templates should be used to declare a mix operation.
84 4 Christopher Green
85 4 Christopher Green
Such an operation may be specified by providing:
86 4 Christopher Green
87 4 Christopher Green
# an @InputTag@ specifying which secondary products should be mixed;
88 4 Christopher Green
# an optional instance label for the mixed product (defaulting to the instance label of the incoming product if unspecified); and
89 4 Christopher Green
# a callable mixer such as:<pre><code class="cplusplus">bool mixfunc(std::vector<PROD const *> const &,
90 6 Christopher Green
             PROD &,
91 6 Christopher Green
             PtrRemapper const &);</code></pre>As the user may prefer, the mixer may take the form of:
92 4 Christopher Green
** an arbitrarily-named free function, or
93 4 Christopher Green
** a function object whose @operator()@ must have the correct type (see @mixfunc@ above), or
94 4 Christopher Green
** an arbitrarily-named member function of any class. In each case, the mixer must have the same type (i.e., the same return type and the same parameter types) illustrated by @mixfunc@ above. The return value of the mix function is taken to indicate whether the product should be placed in the event.
95 4 Christopher Green
   For free functions, function objects, and pre-bound member functions, the product type template argument need not be specified as it can be deduced from the signature of the provided function.
96 10 Walter E Brown
   If the provided function is a member function it may be provided bound to the object upon which it is to be called by the user (in which case it is treated as a free function by the registration method) or by specifying the member function followed by the object to which it should be bound (in which case the bind will be done for the user). In this latter case the template argument specifying the product type need *not* be specified usually as it may be deduced from the signature of the provided function. If one specifies an overload set however (e.g., in the case where a class has several @mix()@ member functions, each with arguments indicating a different product) then the template argument must be specified in order to constrain the overload set to a single function.
97 5 Christopher Green
# An optional @bool@, "outputProduct," defaulting to, "true." A false value for this parameter indicates that the mix product will *never* be put into the event and should therefore not be declared by the @MixHelper@. If the mix operation so registered ever returns true an exception will be thrown.
98 4 Christopher Green
99 4 Christopher Green
@declareMixOp<>()@ may be called with any of the following argument combinations:
100 4 Christopher Green
101 4 Christopher Green
# Provide an @InputTag@ and a mixer that is a free function or function object (or pre-bound member function).
102 4 Christopher Green
# Provide an @InputTag@, an output instance label, and a mixer that is a free function or function object (or pre-bound member function).
103 4 Christopher Green
# Provide an @InputTag@, a mixer that is a non-@const@ member function (of any class), and an object to which that member function should be bound.
104 4 Christopher Green
# Provide an @InputTag@, an output instance label, a mixer that is a non-@const@ member function (of any class), and an object to which that member function should be bound.
105 4 Christopher Green
# As 3, but with a @const@ member function.
106 4 Christopher Green
# As 4, but with a @const@ member function.
107 1 Christopher Green
108 1 Christopher Green
109 5 Christopher Green
h3. @CollectionUtilities.h@: @concatContainers()@ and @flattenCollections()@.
110 4 Christopher Green
111 5 Christopher Green
This header provides two utilities for dealing with the merging of collections:
112 4 Christopher Green
113 5 Christopher Green
h4. @concatContainers()@.
114 4 Christopher Green
115 5 Christopher Green
<pre><code class="cplusplus">template <class CONTAINER>
116 5 Christopher Green
void
117 5 Christopher Green
concatContainers(CONTAINER & out, CONTAINER const & in);</code></pre>
118 4 Christopher Green
119 5 Christopher Green
Append the contents of container @in@ to those of container @out@.
120 4 Christopher Green
121 5 Christopher Green
h4. @flattenCollections()@.
122 5 Christopher Green
123 5 Christopher Green
<pre><code class="cplusplus">template <class COLLECTION>
124 5 Christopher Green
void
125 5 Christopher Green
flattenCollections(vector<COLLECTION const *> in, COLLECTION & out);
126 5 Christopher Green
127 5 Christopher Green
template <class COLLECTION, class OFFSETS>
128 5 Christopher Green
void
129 5 Christopher Green
flattenCollections(vector<COLLECTION const *> in,
130 5 Christopher Green
                   COLLECTION & out,
131 5 Christopher Green
                   OFFSETS & offsets);</code></pre>
132 5 Christopher Green
133 5 Christopher Green
Given a sequence of collections by @const *@ accumulate them into @out@, possibly storing the offsets (for use in remapping @Ptr@ into these collections) for use by @PtrRemapper@ (see below). *N.B.* if @COLLECTION@ is a @PtrVector<T>@ then the extra check is made that the individual @PtrVector<T>@ are compatible (ie point to the same product). If this is not the case, you should merge their underlying collections into a single merged product first (in a separate merge operation) and then use @PtrRemapper@ to re-base and merge the @PtrVector<T>@ collections.
134 1 Christopher Green
135 1 Christopher Green
h3. @PtrRemapper@. 
136 6 Christopher Green
137 6 Christopher Green
If at some point you have one or more @Ptr@ objects pointing to collections which have been merged, they will need to be re-based against the merged collection. This will mean changing the underlying @ProductID@ of the pointer-to product and also adjusting the offset by an amount related to the pointed-to collection's position within the new, merged collection. It goes without saying therefore, that the underlying collections should be merged before attempting to re-base any @Ptr@ into them.
138 6 Christopher Green
139 6 Christopher Green
Each merge function, every event, is provided via its interface with a reference to a @PtrRemapper@ functor which is primed to be capable of re-basing @Ptr@ objects in the current event. @PtrRemapper@ has several @operator()@ templates that accomplish this. Given @PtrRemapper const & remamp@:
140 6 Christopher Green
141 10 Walter E Brown
# Remap a single @Ptr@.<pre><code class="cplusplus">Ptr<A> newPtr(remap(oldPtr, offset));</code></pre>
142 10 Walter E Brown
# Remap a single @PtrVector@.<pre><code class="cplusplus">PtrVector<A> newPV(remap(oldPV, offset));</code></pre>
143 10 Walter E Brown
# Remap a compatible collection (including @PtrVector@) of @Ptr@ providing begin, end iterators (This will also remap a compatible collection of @PtrVector@, but not of PtrVector const * -- for the latter, see 4-10.)<pre><code class="cplusplus">PtrVector<A> newPV;
144 6 Christopher Green
      remap(oldPV.begin(),
145 6 Christopher Green
            oldPV.end(),
146 6 Christopher Green
            std::back_inserter(newPV),
147 6 Christopher Green
            offset);</code></pre>
148 10 Walter E Brown
# Remap and flatten a set of products which are containers of @Ptrs@ (which includes @PtrVector@).<pre><code class="cplusplus">remap(in, out, offsets);</code></pre>where offsets is likely calculated by the appropriate call to @art::flattenCollections<>()@. See source:art/Persistency/Common/CollectionUtilities.h for details.
149 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which may be obtained from a component of the provided product. Provide a free function of the correct signature to return a reference to the container of @Ptrs@ given a secondary product, e.g.:<pre><code class="cplusplus">PtrVector<B> const &myfunc(A const *prod) {
150 6 Christopher Green
  return prod->myBs();
151 6 Christopher Green
}
152 6 Christopher Green
153 6 Christopher Green
remap(in, out, offsets, &myfunc);</code></pre>
154 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which may be obtained from a component of the provided product. Provide the name of a member function of the provided product which is an accessor for the container (taking no arguments).<pre><code class="cplusplus">remap(in, out, offsets, &A::myBs);</code></pre>
155 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which may be obtained from a component of the provided product. Provide the name of a member datum of the provided product which is the container.<pre><code class="cplusplus">remap(in, out, offsets, &A::myBs);</code></pre>
156 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which is a component of the provided product using the provided accessor member function of a class which is not the product.<pre><code class="cplusplus">class Aprocessor {
157 6 Christopher Green
public:
158 6 Christopher Green
  B const &myBs(A const *);
159 6 Christopher Green
};
160 6 Christopher Green
161 6 Christopher Green
Aprocessor myAp;
162 6 Christopher Green
163 10 Walter E Brown
remap(in, out, offsets, Aprocessor::myBs, myAp);</code></pre>Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:<pre><code class="cplusplus">const_cast<Aprocessor &>(myAp);</code></pre>
164 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which is a component of the provided product using the provided @const@ accessor member function of a class which is not the product.<pre><code class="cplusplus">class Aprocessor {
165 6 Christopher Green
public:
166 6 Christopher Green
  B const &myBs(A const *) const;
167 6 Christopher Green
};
168 6 Christopher Green
169 6 Christopher Green
Aprocessor myAp;
170 6 Christopher Green
171 10 Walter E Brown
remap(in, out, offsets, Aprocessor::myBs, myAp);</code></pre>Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:<pre><code class="cplusplus">const_cast<Aprocessor const &>(myAp);</code></pre>
172 10 Walter E Brown
# More general version of 5-9 that takes a final argument which is of arbitrary type provided it or its @operator()@ has the correct signature. The drawback is that one of the template arguments (@CONT@, specifying the type of the collection of Ptrs you wish to remap) is not deducible, meaning that instead of:<pre><code class="cplusplus">remap(...);</code></pre>one must type, for example:<pre><code class="cplusplus">remap.operator()<std::vector<art::Ptr<B> > >(...)</code></pre>Therefore, 4-9 are the recommended signatures for straightforward client code -- this one is provided for maximum flexibility.
173 7 Christopher Green
174 7 Christopher Green
h2. Examples.
175 7 Christopher Green
176 7 Christopher Green
source:test/Integration/MixFilterTest_module.cc
177 7 Christopher Green
178 7 Christopher Green
"mu2e/Offline/EventMixing/src/MixMCEvents_module.cc":http://cdcvs0.fnal.gov/cgi-bin/public-cvs/cvsweb-public.cgi/mu2e/Offline/EventMixing/src/MixMCEvents_module.cc?rev=HEAD&content-type=text/x-cvsweb-markup&only_with_tag=MAIN