8/26/2011 10am meeting with Kirk and Nick from NIU to discuss development projects involving the new NIU GPU-centric cluster.

Email notes

From Jim:
I told some of you already that Marc and I talked with Nick Karonis, the Computer Science Department chair at NIU
about getting involve in some way with their new GPU-centric cluster (60 nodes + 120 GPUs). This came up
in the context of the internship program we are part of at NIU (the RDI program). I believe Nick's goal is to
participate in some sort of development project that can, in the end, be exercised on the NIU cluster and have
as associated publication about the problem. He offered to help educate students on use of MPI and other parallel
processing tools.

Panagiotis mentioned last week that it would be useful to talk with them. It looks like Friday morning is good
for them and for us (Marc and I). Please let me know if you can be available to talk with Nick and one of the
involved professors. We can meet in person or our the phone.

From Nick:
I think the discussion can be a little of both - high level and detailed - depending on
the state of project(s), meaning, I'd be interested to hear (high level) about any project
that you think might be good to work on together, but I would also definitely want
to dive down into as much detail as possible on 1 or 2 of the projects.

By the end of the meeting it would be good to have a (high level) list of projects that
we might work on together coupled with a detailed plan (even if its just work to
test the waters for feasibility) of specific action items for 1 or 2 of those projects.
If we think time may be short or you think the meeting membership may grow too
large, then I would sacrifice the broader high level discussion and focus exclusively
on deeply understanding 1 or 2 projects towards the goal of developing action item list(s).

FNAL areas of interest

DAQ Experiments

We are in need of a framework for evaluating DAQ configurations and determining design parameters for compact, high-throughput systems. This framework will use MPI (probably MPI2 one-sided calls would be best if they worked as promised, along with the reliability options so nodes can drop out and come back without bring down the whole application). The framework also included converting our existing single-threaded event reconstruction/filtering framework to a multithreaded application. It could also involve choosing one or two filtering algorithms and implementing them on the GPUs.

The experiments for which this technology might be interesting will produce data rates of order 100GB/s. We would like to understand how much processing can be done on data collected at this rate. For example, we could try using a 3-d Hough transform to identify tracks in the data, simulating the use of this algorithm in a trigger system.

We may also be interested in investigating MPI-IO. This may or may not be of direct interest to a DAQ system; however, if we have an MPI-capable framework, we'd like to be able to use parallel i/o in offline processing. The libraries upon which we base our current i/o system are not capable of parallel i/o, and in fact are not thread-safe.

Accelerator modeling

Move some of the OpenMP-based development here, with an eye towards multiple GPU use for parts of the algorithm. This is an extension of the current program.

Geant4 experiments for future GPU use

We've been brainstorming about possible ways to reorganize the processing that occurs within Geant4. We are most interested in breaking off a chunk that can be computed within a GPU. Such a break could require substantial reorganization of the Geant4 guts. A simpler demonstration might be worthwhile.

The GPU-targeted algorithms might include:

  • physics algorithms which are current table-based but could be replaced by direct calculation on a massively parallel system,
  • physics algorithms for which the large GPU memory might allow table-based algorithms to be more efficient than current algorithms,
  • computational geometry tasks, which use much memory and much of the computing time in current HEP simulations,
  • integrating equations of motion for large numbers (hundreds or thousands) of particles simultaneously.

Computational Cosmology

Discussions with this group have just begun. There looks to be interesting work here, but it is too soon to pin-point something.

NIU areas of interest


  • What sort of people and experience do you have to offer?
  • What departments do you have connections with?
  • What are your research interests?
  • What is your preference for involving a student in the RDI program?
  • Are there other students you would like to involve in this effort?