Building Containers at Fermilab

Fermilab have provided ANNIE with a build server capable of building Singularity and Docker images: annieimagegpvm01.

Getting Registered

New users who wish to use it to build containers may need to be registered as a user first: ask someone with access to the repoadmin account to register you on the server. Administrators are reminded that the repoadmin account is to be used only for registering new users, and should not be used for building containers. Instructions on registering users is provided when logging in to the repoadmin account.

Building a Container

Once you are registered you may use annieimagegpvm01 to run privileged Singularity (or Docker) to build containers.

Building a Docker Container

Docker containers may be built and pushed to/pulled from Fermilab's own registry
For more information, see the Build Service redmine page:
Disclaimer: this has not yet been attempted. Be prepared for some trial and error, and please update this wiki page with any experiences.

Building a Singularity Container

To be accessible for use on anniesl7gpvm01, containers should be written to the /containers path. This is in fact a link to the /annie/data/containers directory, mounted to allow containers therein to be executed (/annie/data is mounted noexec). It is worth bearing in mind that space in this area may become limited in the future.

Moreover, only the anniepro account has write permission to the /containers directory. To build an image accessible from anniesl7gpvm01, the build must be done from the anniepro account.

The build process on the annie build server is fairly typical, but there are some caveats. By default Singularity uses an area within $HOME for caching layers and temporary storage. Due to the security risks of providing privileged Singularity (or Docker), the /nashome path (i.e. $HOME) is not accessible†, so alternative paths must be provided for the build cache and temp directories. Similarly Singularity writes some temporary files to /tmp, but there is often insufficient space provided at this location for successful build, so an alternative area must be provided. Finally when building an image from Docker Hub, singularity will try to pull docker login credentials from a configuration file somewhere in $HOME, and fall over when this path isn't readable. This aborts the build, despite the fact that no credentials are needed! By passing the --docker-login argument, docker credentials will instead be requested at the prompt: anything may be entered as anniesoft isn't a private repository, but this step is needed all the same.
The last thing to note is that unprivileged singularity cannot directly run the default squashfs image (.simg) container format generated by Singularity build. While unprivileged Singularity can expand such images into a sandbox for use, it makes sense to build the container as a persistent sandbox directory from the start, by passing the --sandbox option.

The resulting build process for building the sandbox directory doodad_sandbox from the latest ToolAnalysis image is:

me@localhost:~$ kinit macguffin
me@localhost:~$ ssh -AK
-bash-4.2$ cd /containers
-bash-4.2$ export SINGULARITY_CACHEDIR=$PWD/cachedir
-bash-4.2$ export SINGULARITY_TMPDIR=$PWD/tmpdir
-bash-4.2$ export TMPDIR=$PWD/tmpdir
-bash-4.2$ export SINGULARITY_LOCALCACHEDIR=$PWD/lcachedir
-bash-4.2$ singularity build --docker-login --sandbox doodad_sandbox docker://anniesoft/toolanalysis:latest

Note that the -AK arguments (user agent and kerberos ticket forwarding) are required to ssh into the anniepro account.

† On logging in it is normal to see the warnings:

mkdir: cannot create directory ‘/nashome’: Permission denied
-bash: /nashome/m/macguffin/.bash_profile: Permission denied