ANNIE-GPVM Singularity Containers


Singularity is a container application, similar to and compatible with Docker, specifically designed to meet the security requirements of the High Performance Computing sector. Recently, Fermilab and the Open Science Grid (OSG) have started to support Singularity on Fermilab GPVMs, FermiGrid worker nodes and OSG Worker Nodes.
Singularity may run privileged, when the Singularity executable has suid enabled (similar to Docker), or unprivileged through the use of user namespaces on sufficiently new kernels. Privileged singularity is required to build containers, but unprivileged singularity may run existing containers.

At the time of writing (24/07/19) ANNIE's standard GPVMs (anniegpvm01 and anniegpvm02) run SL6, which do not support user namespaces. As a pilot run a new gpvm, anniesl7gpvm01, has been created with SL7 and singularity installed.

Using Unprivileged Singularity

The steps to run singularity are quite straightforward:

me@localhost:~$ kinit macguffin
me@localhost:~$ ssh -AKXY
macguffin@anniesl7gpvm01:/annie/app/users/macguffin$ singularity shell -B/pnfs:/pnfs,/cvmfs:/cvmfs,/annie/data/:/annie/data,/annie/app:/annie/app /containers/toolanalysis_latest_29_08_19_sandbox/
Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin> echo $SINGULARITY_CONTAINER
Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin> whoami
Some things to note:
  • The prompt will change to indicate that you are in a container, and the SINGULARITY_CONTAINER environmental variable will be populated (this can be useful to test whether one is executing from a container).
  • Your user is unchanged, including read and write privileges. Any files you create from within the container will be owned by the host user who invoked Singularity.
  • Your shell location will be unchanged. Paths bind-mounted into the container have permissions as per the user on the host. Files and directories that reside in the container are read-only (the container is immutable).
  • $HOME is bind-mounted in by default. Additional paths may be specified with the -B argument, as a comma-delimited list of hostdir:containerdir. In the example above, /annie/data and /pnfs are mounted to be accessible from the container.
  • X display forwarding will work normally (provided it is enabled for ssh via the -X or -Y switch).

Using ToolAnalysis within Singularity

Since files stored within the container are immutable, the ToolAnalysis software package provided by the container can only be used to run Toolchains of existing tools. To run such a toolchain, the user needs only provided the necessary configuration files that describe your toolchain:
  • Create a directory for the config files
  • Create a 'ToolChainConfig' file to describe ToolChain properties
  • Create a 'ToolsConfig' file to describe the tools in your toolchain
  • Create a config file for each Tool
  • Invoke Analyse with your ToolChainConfig file: e.g.
Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin> cd /ToolAnalysis
Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin> ./Analyse /annie/app/users/macguffin/MyConfigFiles/ToolChainConfig

For development work it is usually necessary to edit or build new tools. In this case, use the container to provide all the necessary dependencies that will not change. Packages such as python are provided within the container's standard system directories. ANNIESoft dependencies are provided by the ToolDAQ folder.

macguffin@anniesl7gpvm01:/annie/app/users/macguffin$ git clone MyToolAnalysis
macguffin@anniesl7gpvm01:/annie/app/users/macguffin$ cd MyToolAnalysis
macguffin@anniesl7gpvm01:/annie/app/users/macguffin/MyToolAnalysis$ singularity shell -B/pnfs:/pnfs,/cvmfs:/cvmfs,/annie/data/:/annie/data,/annie/app:/annie/app /containers/toolanalysis_latest_29_08_19_sandbox/
Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin/MyToolAnalysis> ln -s /ToolAnalysis/ToolDAQ ./ToolDAQ
Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin/MyToolAnalysis> source
Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin/MyToolAnalysis> make -f Makefile.Singularity

Minor notes on ToolAnalysis with singularity

  • If you setup singularity as described above, all users that are working in singularity on the cluster will share a common /tmp directory, which might lead to problems when files are not deleted properly and filling up this very limited common space. This is particularly important when working with raw data files as they tend to be rather large. In order to de-couple your singularity /tmp area, you can bind it to a local directory of yours with the binding command -B at the beginning:
    mkdir /annie/app/users/macguffin/local_tmp/
    singularity shell -B/pnfs:/pnfs,/cvmfs:/cvmfs,/annie/data/:/annie/data,/annie/app:/annie/app,/annie/app/users/macguffin/local_tmp:/tmp /containers/toolanalysis_latest_29_08_19_sandbox/
  • If you need up-to-date versions of some ToolDAQ repository for your study and don't have permissions to create a new updated container, you can link a local ToolDAQ directory instead of the ToolDAQ-directory inside of the container. Within this local directory, you can download the latest version of your required software from the ANNIEsoft github page. For example, if you wanted to use a newer version of the MrdTrackLib repository, you could do
    macguffin@anniesl7gpvm01:/annie/app/users/macguffin$ mkdir local_ToolDAQ
    macguffin@anniesl7gpvm01:/annie/app/users/macguffin$ cd local_ToolDAQ
    macguffin@anniesl7gpvm01:/annie/app/users/macguffin/local_ToolDAQ$ git clone MrdTrackLib
    macguffin@anniesl7gpvm01:/annie/app/users/macguffin/local_ToolDAQ$ cd ../MyToolAnalysis/
    singularity shell -B/pnfs:/pnfs,/cvmfs:/cvmfs,/annie/data/:/annie/data,/annie/app:/annie/app,/annie/app/users/macguffin/local_tmp:/tmp /containers/toolanalysis_latest_29_08_19_sandbox/
    Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin/MyToolAnalysis> ln -s /annie/app/users/macguffin/local_ToolDAQ/ ToolDAQ

    Of course you still need to make sure that you have a complete ToolDAQ directory. Any directories that you don't have in your local ToolDAQ should be linked to the directory in the singularity container. This especially applies to software like (root, boost, log4cpp, Pythia6Support, zeromq, Generator-R) that you wouldn't want to update anyway:
    Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin/MyToolAnalysis> cd ToolDAQ
    Singularity toolanalysis_latest_29_08_19_sandbox:/annie/app/users/macguffin/MyToolAnalysis> ln -s /ToolAnalysis/ToolDAQ/boost_1_66_0 boost_1_66_0