Feature #24546: Add the ability to use CVMFS on sites that do not provide local installations - GlideinWMS - Fermilab Redmine

Feature #24546

Add the ability to use CVMFS on sites that do not provide local installations

Added by Marco Mambelli 7 months ago. Updated about 2 months ago.

Status:

Work in progress

Priority:

Normal

Assignee:

Namratha Urs

Category:

Target version:

v3_7_3

Start date:

06/29/2020

Due date:

% Done:

95%

Estimated time:

(Total: 0.00 h)

Stakeholders:

CMS, OSG

Duration:

Description

CVMFS is a write-once read anywhere distributed file system based on HTTP. It is very useful for VO jobs:

singularity images are hosted on it
VO software is frequently hosted there
sometimes data (like calibration data) is hosted there

Several sites do provide a local installation of CVMFS. Some though, especially HPC resources, do not.
cvmfsexec is a tool that allows having CVMFS also on sites where it is not available.
There are 4 different ways it can provide CVMFS, different in terms of requirements, robustness, and ease of use.

mountrepo/umountrepo only
- requires fusermount; mounts in user space
- map /cvmfs in container with singularity –bind (not run from cvmfs because of path)
cvmfsexec on RHEL 7.6 or 7.7
- requires fusermount and additionally unprivileged user namespaces enabled
- maps /cvmfs without singularity, can run singularity under it
- unmounts repos on exit, but not with kill -9
cvmfsexec on RHEL >= 7.8
- no fusermount needed and cleans up mounts even with kill -9
- still needs unprivileged user namespaces enabled; that is the default on RHEL 8
singcvmfs on any system with singularity >= 3.4.0
- drop-in replacement for singularity; set environment variable with cvmfs repos to mount
- requires container image to already be present (not read from cvmfs)
- requires setuid-root singularity except when RHEL >= 7.8 and and singularity >= 3.6.0

Consider also

package building (at reconfig/upgrade)
proxy search and setup (WPAD)

The goal is to ship cvmfsexec and use it if needed selecting the most reliable option to provide CVMFS given the worker node setup.
Possibly the download of cvmfsexec should be optional to reduce the downloads for sites where it is not needed.

After installing CVMFS, the next step is to use it to start singularity in unprivileged mode and have the user jobs run in singularity w/ CVMFS available, see [#24407]

Here some more info about cvmfsexec:

Some suggestions:

use shell (bash) or python (code compatible w/ both 2 and 3). These are the most portable options
The ticket about condor_chirp [#21711] could be a good reference since the task was similar. The python infrastructure now is in place and available
Possible steps
- First test the scripts on a worker node locally (copying the environment of a glidein)
- Then use the <file> attribute in the configuration to add files to download from the Frontend or Factory
- Last make it automatic, including the needed files in the default lists

Subtasks

Related issues

History

#1 Updated by Marco Mambelli 7 months ago

Description updated (diff)

#2 Updated by Marco Mambelli 7 months ago

Blocks Feature #24407: Support CVMFS and Singularity when none of the 2 is available on the worker node added

#3 Updated by Marco Mambelli 7 months ago

Some ideas about the script

This feature could be implemented as a single script or as multiple one (probably a single one w/ functions is better).
The code for this feature could invoke cvmfsexec or mountrepo as needed or include code from those scripts. Here probably invoking the scripts downloaded in a separate tarball makes maintenance easy.

Supposing a single script is invoked by the glidein and a tarball downloaded w/ the needed auxiliary files, then this feature should include 2 scripts and modification to the GlideinWMS code:

script 1: executed in the glidein environment as one of the test/setup scripts at startup (custom script), see https://glideinwms.fnal.gov/doc.prd/factory/custom_scripts.html
script 2: used during the Factory setup phase to build the tarball w/ auxiliary files
modification to GWMS code to include these downloads by default and to build the aux files tarball during reconfig/upgrade

Incremental deliverables:

Prototype
- Testing in the WN environment
- Running the commands by hand, place the cvmfsexec and montrepo there
- Prepare script 1 v0 to automate the checks and the steps to mount CVMFS
- The aux files tarball is built by hand (take note of the commands you use)
- Test on RHEL7 w/ unprivileged userns, on RHEL7 w/o unprivileged userns, [optionally on RHEL6, on RHEL8 (docker container)]
Working feature
- The script 1 is working as a regular custom script (parameters form glidein_config)
- The aux files tarball is built by hand (take note of the commands you use)
- The aux files tarball and the script 1 are added to the <files> section in the factory configuration
Integrated into GlideinWMS
- GlideinWMS code is modified to run the tarball preparation during reconfig/upgrade using script 2
- GlideinWMS code is modified to add the script 1 and tarball to the default list of uploads

Some more details about script 1:

detecting the platform (rhel7, rhel8, rhel6, other, unpriv. userns enabled yes/no (think of the steps to do, test the commands first, do a script outputting different strings/numbers depending on what is detected, make a function of it to include in script 1)
detect is CVMFS is already available and mount repos if it is (list of repos can come from an environment variable or command-line argument, use same steps as above)
mounting CVMFS using mountrepo (list of repos can come from an environment variable or command-line argument, use same steps as above)
mounting CVMFS using cvmfsexec (list of repos can come from an environment variable or command-line argument, use same steps as above)
integrate the parts above
think of anything else useful and add it (improve output/error/debug messages, use info/warn/error/debug functions, ...)
test on platforms in order

These are all suggestions.
Feel free to make differently.
Deliverables should all be recorded (text in this ticket, a new ticket, or attached or linked document, code in git).

The first step will be describing w/ words the steps more in detail.
I will review that and you can continue w/ the coding.
If you have already an idea and prefer to submit directly the code, feel free to do that.

#4 Updated by Namratha Urs 7 months ago

% Done changed from 0 to 20
Status changed from New to Work in progress

Prototype: Prepare a script that automates system checks and performs required steps to mount CVMFS on a worker node environment.

Script 1:

Perform required system checks (OS variant, kernel version, unprivileged userns, and others) and store the results in variables for later use
OS, OS_VARIANT, KERNEL_NUM, KERNEL_PATCH_NUM
CVMFS_MNT, IS_CVMFS_MNT
UNPRIV_USERNS_1, IS_UNPRIV_USERNS_1
UNPRIV_USERNS_2, IS_UNPRIV_USERNS_2
FUSERMOUNT, IS_FUSERMOUNT
FUSE_INSTALLED, IS_FUSE_INSTALLED
USR_IN_FUSE_GRP, IS_USR_IN_FUSE_GRP

Log the above variables (print to stderr) for easy debugging (collect info about the nodes that can be useful for troubleshooting and gather stats about what is out there)

Check if CVMFS is mounted on the worker node using IS_CVMFS_MNT
if IS_CVMFS_MNT is 0
    mount required CVMFS repos with appropriate output/error messages (using mount command)
else
    if OS = rhel
        if OS_VARIANT = 7.8 with KERNEL_NUM >= 3.10.0 and KERNEL_PATCH_NUM >= 1127
            # check for unprivileged user namespaces
            if IS_UNPRIV_USERNS_1 = 0 and IS_UNPRIV_USERNS_2 = 0
                # unprivileged userns is enabled
                use cvmfsexec to mount CVMFS repositories
              else         # i.e. either IS_UNPRIV_USERNS_1 != 0 or IS_UNPRIV_USERNS_2 != 0 or both
                # unprivileged userns is not enabled
                use mountrepo/umountrepo to mount CVMFS repositories
        elif OS_VARIANT = 7.6
            # check for unprivileged user namespaces
            if IS_UNPRIV_USERNS_1 = 0 and IS_UNPRIV_USERNS_2 = 0
                if IS_FUSERMOUNT = 0
                    # unprivileged userns is enabled and fusermount is available
                    use cvmfsexec to mount CVMFS repositories             
                else
                    # unprivileged userns is enabled and fusermount is not available
                    ----- THINK ABOUT THIS CASE! -----
            else         # i.e. either IS_UNPRIV_USERNS_1 != 0 or IS_UNPRIV_USERNS_2 != 0 or both
                # unprivileged userns is not enabled
                use mountrepo/umountrepo to mount CVMFS repositories
        elif OS_VARIANT >= 6 and OS_VARIANT < 7
            if IS_UNPRIV_USERNS_1 != 0 and IS_UNPRIV_USERNS_2 != 0
                # no unprivileged userns
                if IS_FUSE_INSTALLED = 0 and IS_USR_IN_FUSE_GRP = 0
                    # FUSE rpm is installed and the user is in 'fuse' group
                    use mountrepo/umountrepo to mount CVMFS repositories
                else         # either FUSE is not installed or user not in 'fuse' group or both
                    ----- THINK ABOUT THIS CASE! -----
        elif OS_VARIANT = 8 and KERNEL_NUM >= 4.18
            # unprivileged user namespaces are enabled (by default)
            use cvmfsexec to mount CVMFS repositories
    else         # OS != rhel (any non-rhel OS)
        ----- THINK ABOUT THIS CASE! -----

Note that the prototype includes a high-level design to start with and will undergo incremental changes (more descriptions about the low-level details when it comes to the process of mounting the CVMFS repositories, use of exec command, scenarios which need further thought etc.) throughout the development phase. Changes to the prototype design will be periodically updated in this thread to improve the prototype over time.