MPI mini-apps

Configuration and multi-process management

Jesse has started to look into what it will take to start an ensemble of art processes on Cori. I gave him example broadcast and scatter-gather code that can be used to manager sending out art configuration information during start-up (both common and process-specific configurations).

The initial goal for a mini-app is to launch two nodes-worth of art processes, each with a common and unique configuration, run the configuration and produce output from the standard ROOT output module (one file per process written into the global file system).

Here are the tasks:

  1. modify the art main program to contain the MPI code necessary to broadcast a common configuration (FHiCL document). demonstrate with two or more ranks.
  2. modify previous program to generate a unique configuration for each process that is started (requires adding the scatter code for the unique part of the configuration).
  3. See Docker use page for next series of test we want to run on Cori, using the art main program developed in the previous step. We want to first run the simple int/double producer test.

Next steps

Next we want to create a new art output module that will take the int/double product (produced in the test) and convert them to data that can be sent using the MPI send calls. Eventually we will likely want to be using the one-sided communication functions (get/put) or async communication functions (isend/irecv). The data will be sent to an aggregation process. For the test application, this can just be a specialized functions that writes the event data buffers directly to a disk file. Soon after, this should likely be an art input module so that the art data products can be reconstituted and written to output using the standard ROOT output module. This does mean that MPI will need to run different flavors of executable. Artdaq does this.