Project

General

Profile

Product Mixing » History » Version 13

Christopher Green, 02/02/2017 02:29 PM

1 1 Christopher Green
h1. Product Mixing
2 1 Christopher Green
3 13 Christopher Green
[ The product mixing feature was significantly extended for art version:2.06.00, and the current version of this page reflects those improvements. For the earlier documentation, please see "Product_Mixing@12":https://cdcvs.fnal.gov/redmine/projects/art/wiki/Product_Mixing/12. ]
4 13 Christopher Green
5 1 Christopher Green
h2. Overview.
6 1 Christopher Green
7 13 Christopher Green
*Product Mixing* is our label for the ability to take a feed from a source of secondary events, and combine the products (usually collections) from multiple events into a single (possibly different type of) product or collection to be placed in the primary event. The motivation is of course obvious: background (e.g., cosmic) events or pile-up. Since art version:2.06.00, this functionality has been extended to subrun and run products, although with certain caveats which are discussed later.
8 1 Christopher Green
9 1 Christopher Green
This feature is implemented by providing a module template, @MixFilter@ as a skeleton to handle the low-level interaction with the primary event and secondary files, including the collection of multiple products from different secondary events and the registration of outgoing products. The user must provide a "detail" class implementation which has some mandatory and some optional members in order to register necessary products at the right time and mix the prepared groups of incoming products. The @MixFilter@ will take care of reading the secondary data, either sequentially or randomly and cycling through multiple secondary input files as necessary.
10 1 Christopher Green
11 1 Christopher Green
In order to make the mixing task easier, we have provided several auxiliary utilities which will be described later.
12 1 Christopher Green
13 8 Christopher Green
In summary, the @MixFilter@ module will do the following tasks:
14 1 Christopher Green
15 1 Christopher Green
# Construct a @MixHelper@ object for use by the user's detail class, hereafter referred to as @Detail@ (class) or @detail@ (object).
16 10 Walter E Brown
# Construct @detail@, passing the helper and the module's parameter set. At this point, @detail@ should register any mix operations (one per product to be mixed) using @MixHelper::declareMixOp<>()@, and any non-mixing products to go in to the event (e.g., bookkeeping objects) using @MixHelper::produces<>()@.
17 1 Christopher Green
# For each primary event:
18 1 Christopher Green
## Call the optional @Detail::startEvent()@ function if it exists so @detail@ can position itself for the coming event.
19 1 Christopher Green
## Call @Detail::nSecondaries()@ to decide how many secondary events to read.
20 1 Christopher Green
## Decide which secondary events are to be read.
21 13 Christopher Green
## Call the optional @Detail::processEventIDs()@ function in case the @Detail@ class is interested in the sequence of event IDs (e.g., for bookkeeping purposes). The analogous optional @Detail::processEventAuxiliaries()@ function will also be called at this time.
22 12 Christopher Green
## Call the registered mixing operation for each product to be mixed _in the order in which they were declared by_ @detail@, providing the secondary data to be mixed and putting the resultant mixed product into the primary event.
23 1 Christopher Green
## Call the optional @Detail::finalizeEvent()@ function to tidy up and possibly put any bookkeeping products (registered with @MixHelper::produces<>()@) into the primary event.
24 1 Christopher Green
25 1 Christopher Green
The @MixFilter@ module will take care of opening new secondary files as necessary, and will additionally call an optional function @Detail::eventsToSkip()@ as each file is opened (in sequential mode only) if it is desired to have a particular offset into each file to insure against coherent backgrounds in events.
26 1 Christopher Green
27 13 Christopher Green
28 2 Christopher Green
h2. Mix operations: what are they and what do they need to do?
29 3 Christopher Green
30 1 Christopher Green
A mix operation is a function provided by the user which returns a @bool@ and expects three arguments:
31 3 Christopher Green
32 1 Christopher Green
* @vector<PROD const *> const &@ containing the secondary products to be mixed;
33 13 Christopher Green
* @OPROD &@, the output product to be filled by the mix function; and
34 3 Christopher Green
* @PtrRemapper const &@, a helper for use remapping @Ptr@ and friends (see below).
35 3 Christopher Green
36 13 Christopher Green
It may be a free function, a member function of @Detail@ or some other user class, or a function object ("functor") whose @operator()@ has the correct signature (see source:art/Framework/IO/ProductMix/MixTypes.h for a definition of a useful template alias, @art::MixFunc@). This mix function supposed to use the information contained in the secondary products to fill the output product, using the @PtrRemapper@ if appropriate, and return a @bool@ indicating whether the output product should be placed in the event.
37 1 Christopher Green
38 1 Christopher Green
All mix operations must be declared _at construction time_ by @detail@, using the @MixHelper@ provided as a constructor argument.
39 1 Christopher Green
40 13 Christopher Green
Mix operations are always called in the event loop. The @art::MixFilter@ module template and associated utilities take care of registering the primary output product, marshaling the secondary data streams, and obtaining and organizing the products from the secondary data stream for mixing into each primary event. If the mix function returns true, the output product will be written into the primary event.
41 13 Christopher Green
42 13 Christopher Green
For non-event-level products, the mix operation is called for every primary event with the sequence of subrun or run products corresponding to the events chosen for mixing from the secondary stream. The user is responsible for handling duplicates. If data are intended for the primary event, then the provided @OPROD@ is the appropriate place; if data are intended for the primary subrun or run, then the detail object's @endSubRun()@ or @endRun()@ function(s) are responsible for putting those (explicitly declared) products into the primary data stream.
43 13 Christopher Green
44 13 Christopher Green
Note that since @art::Ptr@ may not be used in subrun- and run-level products currently, the @art::PtrRemapper@ provided to non-event-level mix operations is unintialized and should not be used.
45 13 Christopher Green
46 1 Christopher Green
h2. Mandatory interface for @Detail@.
47 1 Christopher Green
48 3 Christopher Green
h3. Constructor.
49 3 Christopher Green
50 9 Christopher Green
<pre><code class="cplusplus">Detail(fhicl::ParameterSet const & ps, art::MixHelper & helper);</code></pre>The constructor should use the provided parameter set to extract any configuration; and the helper to declare mix operations and any bookkeeping products with @declareMixOp<>()@ and @produces<>()@;
51 3 Christopher Green
52 3 Christopher Green
h3. @nSecondaries()@.
53 3 Christopher Green
54 4 Christopher Green
<pre><code class="cplusplus">size_t nSecondaries();</code></pre>Tell @MixFilter@ how many secondaries to read for merging into this primary. If required, a random number engine may be obtained from the @RandomNumberGenerator@ service.
55 3 Christopher Green
56 1 Christopher Green
h2. Optional interface for @Detail@.
57 3 Christopher Green
58 1 Christopher Green
h3. @startEvent()@.
59 4 Christopher Green
60 1 Christopher Green
<pre><code class="cplusplus">void startEvent(art::Event const & evt);</code></pre>Do any per-event initialization of @detail@ here.
61 4 Christopher Green
62 1 Christopher Green
63 3 Christopher Green
h3. @eventsToSkip()@.
64 4 Christopher Green
65 1 Christopher Green
<pre><code class="cplusplus">size_t eventsToSkip();</code></pre>If provided and the @MixFilter@ is in sequential secondary mode, this function will be called at the beginning of every file to ascertain how many events to skip before pulling products to merge.
66 3 Christopher Green
67 4 Christopher Green
68 1 Christopher Green
h3. @processEventIDs()@.
69 1 Christopher Green
70 13 Christopher Green
<pre><code class="cplusplus">void processEventIDs(art::EventIDSequence const & seq);</code></pre>@seq@ is an ordered sequence of @art::EventID@ as defined in source:art/Framework/IO/ProductMix/MixTypes.h. Use as you will.
71 1 Christopher Green
72 1 Christopher Green
73 13 Christopher Green
h3. @processEventAuxiliaries()@.
74 13 Christopher Green
75 13 Christopher Green
<pre><code class="cplusplus">void processEventAuxiliaries(art::EventAuxiliarySequence const & seq);</code></pre>@seq@ is an ordered sequence of @art::EventAuxiliary@ as defined in source:art/Framework/IO/ProductMix/MixTypes.h. Use as you will.
76 13 Christopher Green
77 13 Christopher Green
78 1 Christopher Green
h3. @finalizeEvent()@.
79 1 Christopher Green
80 1 Christopher Green
<pre><code class="cplusplus">void finalizeEvent(art::Event & evt);</code></pre>Write any non-merge products (i.e. those you declared with @MixHelper::produces<>()@) to the event with @evt.put<>()@.
81 1 Christopher Green
82 1 Christopher Green
83 13 Christopher Green
h3. @respondToXXXX()@.
84 13 Christopher Green
85 13 Christopher Green
* <pre><code class="cplusplus">void respondToOpenInputFile(art::FileBlock const & fb);</code></pre>
86 13 Christopher Green
* <pre><code class="cplusplus">void respondToCloseInputFile(art::FileBlock const & fb);</code></pre>
87 13 Christopher Green
* <pre><code class="cplusplus">void respondToOpenOutputFiles(art::FileBlock const & fb);</code></pre>
88 13 Christopher Green
* <pre><code class="cplusplus">void respondToCloseOutputFiles(art::FileBlock const & fb);</code></pre>
89 13 Christopher Green
90 13 Christopher Green
Equivalent to the eponymous functions of a module, called at the appropriate times.
91 13 Christopher Green
92 13 Christopher Green
93 13 Christopher Green
h3. @beginSubRun()@
94 13 Christopher Green
95 13 Christopher Green
<pre><code class="cplusplus">void beginSubRun(art::SubRun const & sr);</code></pre>Analogous to the eponymous module function, providing read-only access to the primary subrun.
96 13 Christopher Green
97 13 Christopher Green
98 13 Christopher Green
h3. @endSubRun()@
99 13 Christopher Green
100 13 Christopher Green
<pre><code class="cplusplus">void endSubRun(art::SubRun & sr);</code></pre>Analogous to the eponymous module function, providing write access to the primary subrun. Products registered for the primary subrun may be @put()@ into the @art::SubRun@ at this time.
101 13 Christopher Green
102 13 Christopher Green
103 13 Christopher Green
h3. @beginRun()@
104 13 Christopher Green
105 13 Christopher Green
<pre><code class="cplusplus">void beginRun(art::Run const & r);</code></pre>Analogous to the eponymous module function, providing read-only access to the primary run.
106 13 Christopher Green
107 13 Christopher Green
108 13 Christopher Green
h3. @endRun()@
109 13 Christopher Green
110 13 Christopher Green
<pre><code class="cplusplus">void endRun(art::Run & r);</code></pre>Analogous to the eponymous module function, providing write access to the primary run. Products registered for the primary run may be @put()@ into the @art::Run@ at this time.
111 13 Christopher Green
112 13 Christopher Green
113 1 Christopher Green
h2. Helper utilities.
114 4 Christopher Green
115 4 Christopher Green
h3. @MixHelper@.
116 4 Christopher Green
117 4 Christopher Green
h4. @produces<>()@.
118 4 Christopher Green
119 13 Christopher Green
Analogous to producer or filter @produces<>()@ calls: declare products to be put into the event, subrun or run which are _not_ direct products of a single mix operation.<pre><code class="cplusplus">template <class PROD>
120 4 Christopher Green
void produces(std::string const & instanceName = std::string());
121 4 Christopher Green
122 1 Christopher Green
template <class PROD, BranchType B>
123 4 Christopher Green
void produces(std::string const & instanceName = std::string());</code></pre>
124 1 Christopher Green
125 1 Christopher Green
h4. @DeclareMixOp<>()@.
126 1 Christopher Green
127 1 Christopher Green
These function templates should be used to declare a mix operation.
128 1 Christopher Green
129 13 Christopher Green
All of the available @declareMixOp()@ function templates have the following template arguments:
130 1 Christopher Green
131 13 Christopher Green
# The BranchType (defaults to art::InEvent). Specify explicitly if you wish to mix subrun or run products, or if you need to specify explicitly any of the other template arguments.
132 13 Christopher Green
133 13 Christopher Green
# The incoming product type (deduced from the provided callable mixer argument).
134 13 Christopher Green
135 13 Christopher Green
# The outgoing product type (deduced from the provided callable mixer argument).
136 13 Christopher Green
137 13 Christopher Green
Mix operations operation may be specified by providing to the appropriate @declareMixOp<>()@ function:
138 13 Christopher Green
139 4 Christopher Green
# an @InputTag@ specifying which secondary products should be mixed;
140 4 Christopher Green
# an optional instance label for the mixed product (defaulting to the instance label of the incoming product if unspecified); and
141 4 Christopher Green
# a callable mixer such as:<pre><code class="cplusplus">bool mixfunc(std::vector<PROD const *> const &,
142 13 Christopher Green
             OPROD &,
143 6 Christopher Green
             PtrRemapper const &);</code></pre>As the user may prefer, the mixer may take the form of:
144 4 Christopher Green
** an arbitrarily-named free function, or
145 4 Christopher Green
** a function object whose @operator()@ must have the correct type (see @mixfunc@ above), or
146 4 Christopher Green
** an arbitrarily-named member function of any class. In each case, the mixer must have the same type (i.e., the same return type and the same parameter types) illustrated by @mixfunc@ above. The return value of the mix function is taken to indicate whether the product should be placed in the event.
147 13 Christopher Green
   For most situations, the product type template arguments need not be specified as they can be deduced from the signature of the provided callable object.
148 13 Christopher Green
   If the provided callable object is a member function it may be provided bound to the object upon which it is to be called by the user (in which case it is treated as a free function by the registration method) or by specifying the member function followed by the object to which it should be bound (in which case the bind will be done for the user). In this latter case the template argument specifying the product types need *not* be specified usually as they may be deduced from the signature of the provided function. If one specifies an overload set however (e.g., in the case where a class has several @mix()@ member functions, each with arguments indicating a different product) then either the template arguments must be specified in order to constrain the overload set to a single function, or the function argument must be @const_cast<>()@ to specify the desired signature.
149 5 Christopher Green
# An optional @bool@, "outputProduct," defaulting to, "true." A false value for this parameter indicates that the mix product will *never* be put into the event and should therefore not be declared by the @MixHelper@. If the mix operation so registered ever returns true an exception will be thrown.
150 4 Christopher Green
151 4 Christopher Green
@declareMixOp<>()@ may be called with any of the following argument combinations:
152 4 Christopher Green
153 4 Christopher Green
# Provide an @InputTag@ and a mixer that is a free function or function object (or pre-bound member function).
154 4 Christopher Green
# Provide an @InputTag@, an output instance label, and a mixer that is a free function or function object (or pre-bound member function).
155 4 Christopher Green
# Provide an @InputTag@, a mixer that is a non-@const@ member function (of any class), and an object to which that member function should be bound.
156 4 Christopher Green
# Provide an @InputTag@, an output instance label, a mixer that is a non-@const@ member function (of any class), and an object to which that member function should be bound.
157 4 Christopher Green
# As 3, but with a @const@ member function.
158 4 Christopher Green
# As 4, but with a @const@ member function.
159 1 Christopher Green
160 1 Christopher Green
161 5 Christopher Green
h3. @CollectionUtilities.h@: @concatContainers()@ and @flattenCollections()@.
162 5 Christopher Green
163 4 Christopher Green
This header provides two utilities for dealing with the merging of collections:
164 5 Christopher Green
165 4 Christopher Green
h4. @concatContainers()@.
166 5 Christopher Green
167 5 Christopher Green
<pre><code class="cplusplus">template <class CONTAINER>
168 5 Christopher Green
void
169 4 Christopher Green
concatContainers(CONTAINER & out, CONTAINER const & in);</code></pre>
170 5 Christopher Green
171 4 Christopher Green
Append the contents of container @in@ to those of container @out@.
172 5 Christopher Green
173 5 Christopher Green
h4. @flattenCollections()@.
174 5 Christopher Green
175 5 Christopher Green
<pre><code class="cplusplus">template <class COLLECTION>
176 5 Christopher Green
void
177 5 Christopher Green
flattenCollections(vector<COLLECTION const *> in, COLLECTION & out);
178 5 Christopher Green
179 5 Christopher Green
template <class COLLECTION, class OFFSETS>
180 5 Christopher Green
void
181 5 Christopher Green
flattenCollections(vector<COLLECTION const *> in,
182 5 Christopher Green
                   COLLECTION & out,
183 5 Christopher Green
                   OFFSETS & offsets);</code></pre>
184 5 Christopher Green
185 1 Christopher Green
Given a sequence of collections by @const *@ accumulate them into @out@, possibly storing the offsets (for use in remapping @Ptr@ into these collections) for use by @PtrRemapper@ (see below). *N.B.* if @COLLECTION@ is a @PtrVector<T>@ then the extra check is made that the individual @PtrVector<T>@ are compatible (ie point to the same product). If this is not the case, you should merge their underlying collections into a single merged product first (in a separate merge operation) and then use @PtrRemapper@ to re-base and merge the @PtrVector<T>@ collections.
186 1 Christopher Green
187 6 Christopher Green
h3. @PtrRemapper@. 
188 6 Christopher Green
189 6 Christopher Green
If at some point you have one or more @Ptr@ objects pointing to collections which have been merged, they will need to be re-based against the merged collection. This will mean changing the underlying @ProductID@ of the pointer-to product and also adjusting the offset by an amount related to the pointed-to collection's position within the new, merged collection. It goes without saying therefore, that the underlying collections should be merged before attempting to re-base any @Ptr@ into them.
190 6 Christopher Green
191 6 Christopher Green
Each merge function, every event, is provided via its interface with a reference to a @PtrRemapper@ functor which is primed to be capable of re-basing @Ptr@ objects in the current event. @PtrRemapper@ has several @operator()@ templates that accomplish this. Given @PtrRemapper const & remamp@:
192 10 Walter E Brown
193 10 Walter E Brown
# Remap a single @Ptr@.<pre><code class="cplusplus">Ptr<A> newPtr(remap(oldPtr, offset));</code></pre>
194 10 Walter E Brown
# Remap a single @PtrVector@.<pre><code class="cplusplus">PtrVector<A> newPV(remap(oldPV, offset));</code></pre>
195 6 Christopher Green
# Remap a compatible collection (including @PtrVector@) of @Ptr@ providing begin, end iterators (This will also remap a compatible collection of @PtrVector@, but not of PtrVector const * -- for the latter, see 4-10.)<pre><code class="cplusplus">PtrVector<A> newPV;
196 6 Christopher Green
      remap(oldPV.begin(),
197 6 Christopher Green
            oldPV.end(),
198 6 Christopher Green
            std::back_inserter(newPV),
199 10 Walter E Brown
            offset);</code></pre>
200 10 Walter E Brown
# Remap and flatten a set of products which are containers of @Ptrs@ (which includes @PtrVector@).<pre><code class="cplusplus">remap(in, out, offsets);</code></pre>where offsets is likely calculated by the appropriate call to @art::flattenCollections<>()@. See source:art/Persistency/Common/CollectionUtilities.h for details.
201 6 Christopher Green
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which may be obtained from a component of the provided product. Provide a free function of the correct signature to return a reference to the container of @Ptrs@ given a secondary product, e.g.:<pre><code class="cplusplus">PtrVector<B> const &myfunc(A const *prod) {
202 6 Christopher Green
  return prod->myBs();
203 6 Christopher Green
}
204 6 Christopher Green
205 10 Walter E Brown
remap(in, out, offsets, &myfunc);</code></pre>
206 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which may be obtained from a component of the provided product. Provide the name of a member function of the provided product which is an accessor for the container (taking no arguments).<pre><code class="cplusplus">remap(in, out, offsets, &A::myBs);</code></pre>
207 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which may be obtained from a component of the provided product. Provide the name of a member datum of the provided product which is the container.<pre><code class="cplusplus">remap(in, out, offsets, &A::myBs);</code></pre>
208 6 Christopher Green
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which is a component of the provided product using the provided accessor member function of a class which is not the product.<pre><code class="cplusplus">class Aprocessor {
209 6 Christopher Green
public:
210 6 Christopher Green
  B const &myBs(A const *);
211 6 Christopher Green
};
212 6 Christopher Green
213 1 Christopher Green
Aprocessor myAp;
214 6 Christopher Green
215 10 Walter E Brown
remap(in, out, offsets, Aprocessor::myBs, myAp);</code></pre>Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:<pre><code class="cplusplus">const_cast<Aprocessor &>(myAp);</code></pre>
216 10 Walter E Brown
# Remap and flatten a set of containers of @Ptrs@ (including @PtrVector@) which is a component of the provided product using the provided @const@ accessor member function of a class which is not the product.<pre><code class="cplusplus">class Aprocessor {
217 6 Christopher Green
public:
218 6 Christopher Green
  B const &myBs(A const *) const;
219 6 Christopher Green
};
220 6 Christopher Green
221 6 Christopher Green
Aprocessor myAp;
222 6 Christopher Green
223 10 Walter E Brown
remap(in, out, offsets, Aprocessor::myBs, myAp);</code></pre>Note: if the compiler complains about an unresolved overload set for this signature, try an explicit:<pre><code class="cplusplus">const_cast<Aprocessor const &>(myAp);</code></pre>
224 13 Christopher Green
# More general version of 5-9 that takes a final argument which is of arbitrary type provided it or its @operator()@ has the correct signature. The drawback is that one of the template arguments (@CONT@, specifying the type of the collection of Ptrs you wish to remap) is not deducible, meaning that instead of:<pre><code class="cplusplus">remap(...);</code></pre>one must type, for example:<pre><code class="cplusplus">remap.operator()<std::vector<art::Ptr<B> > >(...)</code></pre>Therefore, 4-9 are the recommended signatures for straightforward client code -- this one is provided for maximum flexibility and for use internally by signatures 5-9.
225 7 Christopher Green
226 7 Christopher Green
h2. Examples.
227 7 Christopher Green
228 7 Christopher Green
source:test/Integration/MixFilterTest_module.cc
229 7 Christopher Green
230 7 Christopher Green
"mu2e/Offline/EventMixing/src/MixMCEvents_module.cc":http://cdcvs0.fnal.gov/cgi-bin/public-cvs/cvsweb-public.cgi/mu2e/Offline/EventMixing/src/MixMCEvents_module.cc?rev=HEAD&content-type=text/x-cvsweb-markup&only_with_tag=MAIN