Singularity jobs

  • By default jobs invoke Singularity at supported sites;
    default images are the FNAL WN SL6/SL7 images located at:
    • SL6: /cvmfs/
    • SL7: /cvmfs/
  • User can override the Singularity image with the jobsub option
    -l '+SingularityImage=\"/path/to/image\"'
    This only works at a site that has Singularity installed, this does NOT guarantee you will get a singularity container. In general, we recommend that you supply the Singularity image option. The --lines= option is equivalent to -l. Also, be very careful that you have all quotes and backslashes in place exactly as shown here.
  • To ensure Singularity, in addition to supplying the SingularityImage option, add
  • Do NOT use the --OS jobsub option, rather control the OS with
    -l '+SingularityImage=\"/path/to/image\"'
    for example, -l '+SingularityImage=\"/cvmfs/\"' or some other Singularity image that you prefer.
  • If your job can only run in an SL6 environment, whether in Singularity or not, you need to do two things to guarantee it. First:
    that will still give you access to the few worker nodes that are native SL6 and don't support Singularity. Second, choose an SL6 Singularity image to run at the sites that support Singularity:
    -l '+SingularityImage=\"/path/to/some/SL6/image\"'
    If your job only works in a FermiGrid-like SL6 environment, do the above, but drop the ||TARGET.IFOS_Installed=?=\"SL6\" part. Again, supply BOTH the --append_condor_requirements=... and -l '+SingularityImage=...' options.
  • If you are already using --append_condor_requirements, you could also combine them in a single option like this:
    --append_condor_requirements='((TARGET.HAS_SINGULARITY=?=true)&&(whatever else you normally put))'
    --append_condor_requirements='((TARGET.HAS_SINGULARITY=?=true||TARGET.IFOS_Installed=?=\"SL6\")&&(whatever else you normally put))'
    as appropriate.

Recommendation to build custom Singularity images

When building a custom Singularity images, we recommend to base the image on the FNAL WN image.
This will make sure the image has all tools required to interact with Fermilab infrastructure (dCache, SAM, ...).
It is worth to recall that most of the Fermilab tools are available for SL and equivalent OSs.
Docker files used to build FNAL WN images are available here

Publishing custom Singularity images on CVMFS

It is recommended to publish Singularity images on CVMFS for use in grid jobs.
The procedure is documented here

Known site issues with Singularity:

  • NotreDame, MWT2, UChicago and FZU currently have an issue that prevents SL6 job to get set up correctly.
    In order to use those sites for SL6 jobs, it is recommended to add at the beginning of the job script the following command:
    source /cvmfs/