Building Containers at Fermilab¶
Fermilab have provided ANNIE with a build server capable of building Singularity and Docker images:
New users who wish to use it to build containers may need to be registered as a user first: ask someone with access to the
repoadmin account to register you on the server. Administrators are reminded that the
repoadmin account is to be used only for registering new users, and should not be used for building containers. Instructions on registering users is provided when logging in to the
Building a Container¶
Once you are registered you may use
annieimagegpvm01 to run privileged Singularity (or Docker) to build containers.
Building a Docker Container¶
Docker containers may be built and pushed to/pulled from Fermilab's own registry imageregistry.fnal.gov
For more information, see the Build Service redmine page: https://ssiwiki.fnal.gov/wiki/Container_Build_Service_Home.
Disclaimer: this has not yet been attempted. Be prepared for some trial and error, and please update this wiki page with any experiences.
Building a Singularity Container¶
To be accessible for use on
anniesl7gpvm01, containers should be written to the
/containers path. This is in fact a link to the
/annie/data/containers directory, mounted to allow containers therein to be executed (
/annie/data is mounted
noexec). It is worth bearing in mind that space in this area may become limited in the future.
Moreover, only the
anniepro account has write permission to the
/containers directory. To build an image accessible from
anniesl7gpvm01, the build must be done from the
The build process on the annie build server is fairly typical, but there are some caveats. By default Singularity uses an area within
$HOME for caching layers and temporary storage. Due to the security risks of providing privileged Singularity (or Docker), the
/nashome path (i.e.
$HOME) is not accessible†, so alternative paths must be provided for the build cache and temp directories. Similarly Singularity writes some temporary files to
/tmp, but there is often insufficient space provided at this location for successful build, so an alternative area must be provided. Finally when building an image from Docker Hub, singularity will try to pull docker login credentials from a configuration file somewhere in
$HOME, and fall over when this path isn't readable. This aborts the build, despite the fact that no credentials are needed! By passing the
--docker-login argument, docker credentials will instead be requested at the prompt: anything may be entered as anniesoft isn't a private repository, but this step is needed all the same.
The last thing to note is that unprivileged singularity cannot directly run the default squashfs image (
.simg) container format generated by Singularity build. While unprivileged Singularity can expand such images into a sandbox for use, it makes sense to build the container as a persistent sandbox directory from the start, by passing the
The resulting build process for building the sandbox directory
doodad_sandbox from the latest ToolAnalysis image is:
me@localhost:~$ kinit macguffin me@localhost:~$ ssh -AK email@example.com -bash-4.2$ cd /containers -bash-4.2$ export SINGULARITY_CACHEDIR=$PWD/cachedir -bash-4.2$ export SINGULARITY_TMPDIR=$PWD/tmpdir -bash-4.2$ export TMPDIR=$PWD/tmpdir -bash-4.2$ export SINGULARITY_LOCALCACHEDIR=$PWD/lcachedir -bash-4.2$ singularity build --docker-login --sandbox doodad_sandbox docker://anniesoft/toolanalysis:latest
Note that the
-AK arguments (user agent and kerberos ticket forwarding) are required to ssh into the
† On logging in it is normal to see the warnings:
mkdir: cannot create directory ‘/nashome’: Permission denied -bash: /nashome/m/macguffin/.bash_profile: Permission denied