Docker use

We have been experimenting with use of docker images under Shifter on Cori. There will be a separate write-up of the container project in general. The purpose of the project is to ease to movement and installation of the art/larsoft run-time environment on new platforms. Our current test image contains all of art, including its externals. Art can be run within it using Shifter at NERSC.

The current test image is based on Centos6.7, which is compatible with SLF6. The standard installation of art works well with this release. The are currently using the ROOT 6 version of art. We may need to also prepare the ROOT 5 version when we start running experiment code.

Here are the list of tasks we want to do to evaluate the use of this technology and demonstrate that it is working properly.

  1. Test the CentOS/art image
  2. Run simple overhead and throughput test.
  3. Run uBooNE simulation task.
  4. Run g-2 simulation task.

Test CentOS/art image

  • test that the image can be downloaded and launched
  • run "art --help" using Slurm and Shifter on Cori.

This is done

Simple overhead and throughput test

Run art using the int / double producer and the IntProduct and DoubleProduct, along with the ROOT output module on and off. Measure the performance and overhead. Since we do not have a native build ready (unless Lisa's stuff is functioning), it will not be very easy to compare performance. We can run the same program configurations on woof to see what the performance differences are.
I am most interested in the overhead of the framework and the overhead of the output module as compared with not running under a container image.
I am also interested in making sure that we can configure and run more than one process running under a docker image on Cori.

The producer and data product code (source and compiled) will be made available inside the image. I am currently building this test code in a prototype "makestudy" area, and it is ready to be incorporated directly on top of the art base image.

experiment simulation tasks

We have been given a configuration for running uBooNE simulation and a container has been constructed with the latest release. After running this on a local test machine, we can attempt to run this on Cori.

The g-2 image still need to be constructed. Afterwords we should test.


Supplying FHiCL configuration files will be an issue. Shifter will mount the home directory of the NERSC user (global filesystem) into the image when it is booted. This can be a source of FHiCL configuration files. If each program instance needs a unique configuration, then care will need to be taken for organizing and providing these files.
Jesse is looking at adding MPI code to the art main program to handle broadcasting of common configuration to all processes, along with a procedure for sending out process-specific configuration (using scatter functions).