Using Docker with NOvA Software » History » Version 39

« Previous - Version 39/48 (diff) - Next » - Current version
Andrew Norman, 05/18/2018 03:48 PM

Using Docker with NOvA Software

Contact Pengfei or join nova-docker slack channel for any questions.

Step 1. Install OSX Fuse ( if you are using macOS. Skip this step for Linux.

Step 2. Install CVMFS

Follow the instructions at to install CVMFS on macOS or Linux (Window is not supported yet). After installation, please do the following to configure CVMFS properly.

sudo wget 
sudo rm -rf /etc/cvmfs/*
sudo tar zxvf cvmfs.tar.gz; sudo mv cvmfs/* /etc/cvmfs/
sudo mkdir /cvmfs/
sudo mkdir /cvmfs/
sudo cvmfs_config reload
sudo mount -t cvmfs /cvmfs/
sudo mount -t cvmfs /cvmfs/

Alternatively, here is a Wiki page with more detailed instructions.

Step 3. Install Docker

  1. For macOS, refer to
  2. after the installation finished, in Docker Preferences, add “/cvmfs” to the “File Sharing” list, click “Apply & Restart” to restart docker.
  3. For Linux, e.g. Ubuntu please refer to for installation instructions. You can choose a different Linux distribution from the menu on the left side of this page.

Step 4. Run the docker container

docker run --rm -it -v /cvmfs:/cvmfs:cached -v $HOME:/scratch dingpf/slf6.7

Step 5. Setup NOvA software environment, e.g.

source /cvmfs/ \
 -e /cvmfs/  \
 -5 /cvmfs/  \
 -6 /cvmfs/ \
 -r S18-02-25 -b maxopt

Step 6. Get a valid voms proxy

kinit YOUR_USER_NAME@FNAL.GOV # replace YOUR_USER_NAME with your fermilab user name
# kx509 has been installed in the image, this is the recommended way of getting the certificate instead of using cigetcert directly.
# use cigetcert -i "Fermi National Accelerator Laboratory" in case kx509 failed.
voms-proxy-init --rfc --voms=fermilab:/fermilab/nova/Role=Analysis --noregen

Step 7. Make addpkg_svn to work properly

mkdir ~/.ssh
# create ~/.ssh/config as the following (replace YOUR_USER_NAME with your fermilab user name)
 ForwardX11 = no
 GSSAPIAuthentication yes
 GSSAPIDelegateCredentials yes

Step 8. Create a test release and build a package

Create your working directory under /scratch (note only files under mounted path to the container will be kept after shutting down the container.

newrel -t S18-02-25 testrel_s180225
cd testrel_s180225
addpkg_svn CAFAna S18-02-25
make all
# The build will fail due to missing shared library links (we will fix it in the development release)
# you will need to add the following line to “CAFAna/Core/GNUmakefile”
# override CPPFLAGS += -I$(BOOST_INC)
# Add similar thing to CAFAna/XSec/GNUmakefile, the line will be like:

Step 9. Running NOvA EventDisplay.

  1. If you want to open an event display, start the docker container with:
    docker run --rm -it -p 5900:5900 -v /cvmfs:/cvmfs:cached -v $HOME:/scratch dingpf/slf6.7
  2. Then run the following script to start the vnc server in the container:
    /home/me/ &
  3. Once the vnc server is up and running, you can connect to the VNC session from your host machine via any VNC client. The address is vnc://localhost:5900. The password is password.
  4. For mac, you can start a VNC viewer by press Cmd+k in Finder, and connect to vnc://localhost:5900.
  5. In the VNC session, there is an xterm opened for you. You can do your usual software setup with "setup_nova" and get a valid voms proxy. You can use xrdcp to get any file from dCache to your container, for example, if you want to run EventDisplay on this file from one of the official dataset, you will need to do the following:
    # setup nova
    source /cvmfs/ \
     -e /cvmfs/  \
     -5 /cvmfs/  \
     -6 /cvmfs/ \
     -r S18-02-25 -b maxopt
    # get the xrootd file access url
    fpath=`samweb get-file-access-url --schema=xroot`
    # Get a valid voms-proxy
    kinit YOUR_USER_NAME@FNAL.GOV # replace YOUR_USER_NAME with your fermilab user name
    voms-proxy-init --rfc --voms=fermilab:/fermilab/nova/Role=Analysis --noregen
    # Do a xrootd copy of the file to your local disk in the container
    xrdcp $fpath ./
    # start the EventDisplay on this file, the following command need to be run
    # INSIDE the xterm in the VNC session (together with the setup_nova).
    nova -c evd.fcl

Feldman-Cousins Corrections in Docker

If you're crazy enough to want to run the 2017 Analysis Feldman Cousins corrections on a local machine, here's how you would do it. You cannot do batch submissions to the grid this way, but this is to illustrate how you might run a single FC job using the docker image: dingpf/sfl6.7.

Download Prerequisites

The 2017 FC script requires some input root files. These will need to be downloaded to the local machine, and can scp'd from /pnfs/nova/persistent/users/ddoyle/localnova.tar.gz
Unpack this to some /path/to/localnova.

Enter Docker

See previous instructions for setting up Docker on your local machine. Once installed, run:

sudo docker run --rm -it -v /path/to/localnova:/scratch -v /cvmfs:/cvmfs:cached dingpf/slf6.7 

# set environment variable used in FC script

The -v /path/to/localnova:/scratch option mounts the localnova volume to the /scratch directory within the docker. It is important that localnova is mounted here for the code to find required files. Feel free to use this volume as storage as changes are not persistent within the docker.

Setup test release

Create a test release from S18-02-25 then add CAFAna from the FC-at-NERSC branch.

newrel -t S18-02-25 <mylocalfc>
cd <mylocalfc>
srt_setup -a
addpkg_svn -b CAFAna FC-at-NERSC
novasoft_build -t

Run the script


void joint_fit_2017_make_fc_surf(int NPts, int bin, bool nh, int N,
                                 std::string plot)

This take NPts, the number of experiments to throw at a bin, bin which bin to throw to, a bool for specifying mass hierarchy, N a bookkeeping parameter, and a string plot specifying either "ssth23dmsq32" or "deltassth23" contours.


cafe -bq joint_fit_2017_make_fc_surf.C 10 10 true 0 ssth23dmsq32


void joint_fit_2017_make_fc_slice(int NPts, int bin, bool nh, int N,
                                  std::string plot="delta")

The procedure for slices is similar. The options for plot are "delta", "ssth23", or "dmsq32".


cafe -bq joint_fit_2017_make_fc_slice.C 10 10 true 0 ssth23

Feldman-Cousins Corrections in Docker without cvmfs

Follow the following instructions to run Feldman-Cousins Corrections in Docker if you do not have cvmfs in your local system.

# Get the standalone docker image with software and spectrums for running FC corrections
scp .

# Register it with docker daemon
docker load -i dingpf--fc-on-nersc--S18-02-25-maxopt-Ana2017.tar

# Get the script to run the docker image
tar zxvf run_fc.tar.gz
cd run_fc

# run FC experiments
./ --macro=CAFAna/nue/Ana2017/joint_fit_2017_make_fc_slice.C --npoints=2 --bin=1 --hierarchy=true --id=0 --plot=ssth23
./ --macro=CAFAna/nue/Ana2017/joint_fit_2017_make_fc_surf.C --npoints=2 --bin=1 --hierarchy=true --id=0 --plot=ssth23dmsq32

Building docker images for NERSC

Currently the image building procedure includes the following steps:
  1. Prepare a test release of CAFAna locally with the version of code you want;
  2. Run a generic docker container and mount the local test release and cvmfs repo to the container, and build the test release in the container;
  3. Also in the container, compile the CAFAna macro you want to run at NERSC.
  4. Gather files needed by CAFAna package from cvmfs, and pull those files from cvmfs repo to a local directory;
  5. Build the docker image with the local test release and local cvmfs directory;
  6. Push the image to NERSC and put it on Cori or Edison.

I've created scripts to locate and pull files from cvmfs, as well as build the docker image. They are under the following directroy on


Under this directory, you will see:

├── dirs.list
├── Dockerfile_nova-fc-2018:fc-on-nersc:development-maxopt-base
├── Dockerfile_nova-fc-2018:fc-on-nersc:development-maxopt-v0.0
├── image
│   ├── cvmfs
│   └── development
├── libs.list

The test release is under


You should follow the directory structure to make the build script work.

Rebuilding the Test Release

At the top level execute the "" script which will run the following docker command:

IMAGE=<path to image subdir in build directory>
sudo docker run --rm -it -v $IMAGE/development:/development -v /cvmfs:/cvmfs dingpf/slf6.7 /development/ru make CAFAna.all

This is built against the full CVMFS repo.

Now you need to rebuild the CAFAna macro. To do this you need to run the macro once.

IMAGE=<path to image subdir in build directory>
sudo docker run --rm -it -v $IMAGE/development:/development -v $IMAGE/cvmfs:/cvmfs dingpf/slf6.7 /development/ru cafe -br <fullpath in image>/<macro_name> <options for macro>

Example Syntax:

sudo docker run --rm -it -v $IMAGE/development:/development -v /cvmfs:/cvmfs /development/ cafe -bq /development/CAFAna/nue/Ana2018/make_fc_slices_nersc_2018_stats.C 1 0 0 true 0 10  5 ssth23 both false false

Rebuilding the Images and pushing to NERSC

To pull the files from cvmfs, run the following:

./ dirs.list image/cvmfs; ./ libs.list image/cvmfs

To build the image, copy "Dockerfile_nova-fc-2018:fc-on-nersc:development-maxopt-v0.0" into "Dockerfile_nova-fc-2018:fc-on-nersc:development-maxopt-v${NEW_VERSION}", where ${NEW_VERSION}" is your desired version number, and then run

sudo ./ Dockerfile_nova-fc-2018:fc-on-nersc:development-maxopt-v${NEW_VERSION}

This script will build the new image, and tag it. It will prompt you to push the image to NERSC registry at the end. Run the command that it prompts you do run to do the push. This must be done AS ROOT (not sudo):
ksu # become root

# Push the image
docker push

If you forget to increment the version number, the image can be retagged via:

sudo docker images # List the images

# Then Retag with
sudo docker tag ba7c53be347e<New Version>

Once the image is pushed to NERSC private registry, loing to Cori and/or Edison, and run the following to make the image available on the Supercomputers:

# ON CORI Load the Shifter Module 
$ module load shifter-registry

# Login to the Registry
$ shifterimg-beta login username: <NERSC username> password: <Application Token>

# Pull the Image in to the Shifter Registry
$ shifterimg-beta pull

Run the container interactively at for NERSC

Once the image is pulled to Cori, you can do the following to run it in the interactive queue.

dingpf@cori02:~> salloc -N 1  -C haswell -q interactive  -t 04:00:00
salloc: Pending job allocation 12513427
salloc: job 12513427 queued and waiting for resources
salloc: job 12513427 has been allocated resources
salloc: Granted job allocation 12513427
salloc: Waiting for resource configuration
salloc: Nodes nid00050 are ready for job

dingpf@nid00050:~> shifter --volume=$CSCRATCH:/output /development/ cafe -bq -nr /development/CAFAna/nus/Nus18/MakeSurfaceMedian.C fhc th24vsdm41 systs both
Running  cafe -bq -nr /development/CAFAna/nus/Nus18/MakeSurfaceMedian.C fhc th24vsdm41 systs both
** NOvA Common Analysis Format Executor **
root -l -n -b -q /development/CAFAna/load_libs.C /development/CAFAna/nus/Nus18/MakeSurfaceMedian.C+("fhc","th24vsdm41","systs","both")