Project

General

Profile

Module threading types

This page talks only about plugins with the "module" suffix; services, sources, tools, and any other plugins are not discussed here.

As of art 3.00.00, modules can be characterized in two dimensions:

  • workflow behavior: is the module a producer, filter, analyzer or output module? This is the module workflow type, or just module type.
  • processing behavior: is the module shared or replicated across schedules? This is the module threading type.

Whereas determining module workflow type is usually straightforward, figuring out the module threading type can be more difficult. This page discusses the tradeoffs involved in making such a decision. Before reading this page, you must understand the concept of a schedule (see Schedules and transitions).

 Shared modules

art supports shared modules and replicated modules. Shared modules are module instances shared across all schedules, the number of which is specified by the user (see the scheduler configuration section). For example, if a user's job configuration looks like:

services.scheduler: {
  num_schedules: 2
}

physics: {
  producers: {
    m1: { module_type: MySharedModule1 }
    m2: { module_type: MySharedModule2 }
    m3: { module_type: MySharedModule3 }
  }
  tp: [m1, m2, m3]
}

created processing infrastructure could be illustrated like this:

Shared modules see all events.

 serialize<art::InEvent>(...)

One of the reasons for using a shared module is if the library you are using cannot guarantee thread safety. In such a case, it is possible to tell the framework that event-level calls relying on a given library should be serialized:

MyProducer::MyProducer(Parameters const& p)
  : SharedProducer{p}
  // other initializations
{
  produce<MyProduct>();
  serialize<art::InEvent>("NameOfThreadUnsafeResource");
}

The argument given to the serialize function is called the resource name, which is of a type convertible to std::string. Whenever it makes sense, a function should be introduced that, when called, returns the resource name as an std::string. This approach avoids the typographical errors for which string-based interfaces are prone (e.g.):

serialize<art::InEvent>("NameOfThreadUnsafeResource");          // 1. discouraged, but not forbidden
serialize<art::InEvent>(ThreadUnsafeResource::resource_name()); // 2. encouraged
serialize(ThreadUnsafeResource::resource_name());               // 3. equivalent to 2

For resources that have a well-defined type, the type should have a static member function called resource_name(). If there is no obvious type to which a resource name should be attributed (e.g. GENIE), it is encouraged that those libraries introduce a free function along the lines of:

namespace GENIE {
  inline std::string resource_name() { return "GENIE"; }
}

so that a user can make a call like GENIE::resource_name().

Other things to note:

  • Explicitly specifying template argument art::InEvent is optional--the InEvent value is the default.
  • Specifying an empty resource-name list--i.e. 'serialize();'--indicates that the module's event-level function may be called at any point irrespective of any other module, but the module in question still processes one event at a time.
  • Multiple resource names may be specified in the serialize call: serialize(TFileService::resource_name(), "CLHEP_random_engine"). The framework guarantees then places the event-level call in the corresponding queues. It is not until the call is at the front of each queue that the framework invokes the user's member function.

But what does it do?

A call to serialize tells the framework to create a queue associated with each resource name. The event-level calls are serially executed for all modules that register for the same queue. For example, suppose a user has configured modules 'm1' and 'm3' to use the same resource:

MySharedModule1::MySharedModule1(...) { serialize(TFileService::resource_name()); } // c'tor for m1
MySharedModule2::MySharedModule2(...) { serialize(); }                              // c'tor for m2
MySharedModule3::MySharedModule3(...) { serialize(TFileService::resource_name()); } // c'tor for m3

then the m1 and m3 event-level calls would not be invoked at the same time on different events. For module m2, however, no resource name has been provided. If m2 were configured on a different path, it would be possible for m2 to execute in parallel with m1 and m3 since there is no resource that is shared between them.

Suppose, however, that string literals were specified instead of making a function call:

MySharedModule1::MySharedModule1(...) { serialize("TFileService"); } // c'tor for m1
MySharedModule2::MySharedModule2(...) { serialize(); }               // c'tor for m2
MySharedModule3::MySharedModule3(...) { serialize("TfileService"); } // c'tor for m3 (oops: 'f' instead of 'F')

Such a spelling error results in the framework not properly serializing 'm1' and 'm3' calls. This is why the resource_name() function call is strongly preferred.

 Legacy modules

Legacy modules are those that inherit from EDProducer, EDFilter, EDAnalyzer or OutputModule. To guarantee that old workflows still work, all legacy modules are shared modules with maximum serialization enabled. The serialization for legacy modules is enabled implicitly by the framework, which parses all 'serialize' calls and assigns each legacy module to each resource queue.

 async<art::InEvent>()

If you can guarantee that the external libraries the module uses are thread-safe, and that the data member members are used in a thread-safe manner, then the async<art::InEvent>() call may be made in the module's constructor:

MyProducer::MyProducer(Parameters const& p)
  : SharedProducer{p}
  // other initializations
{
  produces<MyProduct>();
  async<art::InEvent>();
}

The async call is an instruction to the framework that the event-level function (produce in this case) may be called concurrently with any other module's member functions. A module that makes the async call is an asynchronous shared module.

An asynchronous shared module is the optimal module for memory and efficiency performance.

 Replicated modules

There may be cases where you can ensure that all external libraries used by a module are thread-safe, but it is rather difficult to make individual data members of your module thread safe (e.g. CLHEP random number engines). In such a situation, a replicated module can be used. A replicated module is one where for a given module configuration, one module instance is create per schedule. Assume that the type of module m2 is changed to a MyReplicatedModule:

services.scheduler: {
  num_schedules: 2
}

physics: {
  producers: {
    m1: { module_type: MySharedModule1 }
    m2: { module_type: MyReplicatedModule }
    m3: { module_type: MySharedModule3 }
  }
  tp: [m1, m2, m3]
}

the processing infrastructure would then look like this:

where separate m2 instances have been created for each schedule. You should choose a replicated module if:

  • the module does not need to see every event
  • the module does not need to create SubRun or Run data products (current limitation of art which will be fixed in a newer version)
  • the external libraries used by the module are thread-safe
  • only module data members are not intrinsically thread-safe

 Comparison of module interfaces