Feature #24546
Add the ability to use CVMFS on sites that do not provide local installations
95%
CMS, OSG
Description
- singularity images are hosted on it
- VO software is frequently hosted there
- sometimes data (like calibration data) is hosted there
Several sites do provide a local installation of CVMFS. Some though, especially HPC resources, do not.
cvmfsexec is a tool that allows having CVMFS also on sites where it is not available.
There are 4 different ways it can provide CVMFS, different in terms of requirements, robustness, and ease of use.
- mountrepo/umountrepo only
- requires fusermount; mounts in user space
- map /cvmfs in container with singularity –bind (not run from cvmfs because of path)
- cvmfsexec on RHEL 7.6 or 7.7
- requires fusermount and additionally unprivileged user namespaces enabled
- maps /cvmfs without singularity, can run singularity under it
- unmounts repos on exit, but not with kill -9
- cvmfsexec on RHEL >= 7.8
- no fusermount needed and cleans up mounts even with kill -9
- still needs unprivileged user namespaces enabled; that is the default on RHEL 8
- singcvmfs on any system with singularity >= 3.4.0
- drop-in replacement for singularity; set environment variable with cvmfs repos to mount
- requires container image to already be present (not read from cvmfs)
- requires setuid-root singularity except when RHEL >= 7.8 and and singularity >= 3.6.0
- package building (at reconfig/upgrade)
- proxy search and setup (WPAD)
The goal is to ship cvmfsexec and use it if needed selecting the most reliable option to provide CVMFS given the worker node setup.
Possibly the download of cvmfsexec should be optional to reduce the downloads for sites where it is not needed.
After installing CVMFS, the next step is to use it to start singularity in unprivileged mode and have the user jobs run in singularity w/ CVMFS available, see [#24407]
Here some more info about cvmfsexec:- https://indico.cern.ch/event/813800/contributions/3830191/attachments/2031830/3401805/PreGDBCvmfsexec20200504.pdf
- https://github.com/cvmfs/cvmfsexec (Dave's repo: https://github.com/DrDaveD/cvmfsexec)
- https://s3.cern.ch/inspire-prod-files-1/1a01a09874305d2abd76b78998463bdc (Abstract: https://inspirehep.net/literature/1796162)
And CVMFS: - https://cernvm.cern.ch/portal/filesystem/quickstart
- https://inspirehep.net/literature/1796162
- https://cernvm.cern.ch/portal/filesystem
- use shell (bash) or python (code compatible w/ both 2 and 3). These are the most portable options
- The ticket about condor_chirp [#21711] could be a good reference since the task was similar. The python infrastructure now is in place and available
- Possible steps
- First test the scripts on a worker node locally (copying the environment of a glidein)
- Then use the
<file>
attribute in the configuration to add files to download from the Frontend or Factory - Last make it automatic, including the needed files in the default lists
Subtasks
Related issues
History
#1 Updated by Marco Mambelli 10 months ago
- Description updated (diff)
#2 Updated by Marco Mambelli 10 months ago
- Blocks Feature #24407: Support CVMFS and Singularity when none of the 2 is available on the worker node added
#3 Updated by Marco Mambelli 10 months ago
Some ideas about the script
This feature could be implemented as a single script or as multiple one (probably a single one w/ functions is better).
The code for this feature could invoke cvmfsexec or mountrepo as needed or include code from those scripts. Here probably invoking the scripts downloaded in a separate tarball makes maintenance easy.
- script 1: executed in the glidein environment as one of the test/setup scripts at startup (custom script), see https://glideinwms.fnal.gov/doc.prd/factory/custom_scripts.html
- script 2: used during the Factory setup phase to build the tarball w/ auxiliary files
- modification to GWMS code to include these downloads by default and to build the aux files tarball during reconfig/upgrade
- Prototype
- Testing in the WN environment
- Running the commands by hand, place the cvmfsexec and montrepo there
- Prepare script 1 v0 to automate the checks and the steps to mount CVMFS
- The aux files tarball is built by hand (take note of the commands you use)
- Test on RHEL7 w/ unprivileged userns, on RHEL7 w/o unprivileged userns, [optionally on RHEL6, on RHEL8 (docker container)]
- Working feature
- The script 1 is working as a regular custom script (parameters form glidein_config)
- The aux files tarball is built by hand (take note of the commands you use)
- The aux files tarball and the script 1 are added to the <files> section in the factory configuration
- Integrated into GlideinWMS
- GlideinWMS code is modified to run the tarball preparation during reconfig/upgrade using script 2
- GlideinWMS code is modified to add the script 1 and tarball to the default list of uploads
- detecting the platform (rhel7, rhel8, rhel6, other, unpriv. userns enabled yes/no (think of the steps to do, test the commands first, do a script outputting different strings/numbers depending on what is detected, make a function of it to include in script 1)
- detect is CVMFS is already available and mount repos if it is (list of repos can come from an environment variable or command-line argument, use same steps as above)
- mounting CVMFS using mountrepo (list of repos can come from an environment variable or command-line argument, use same steps as above)
- mounting CVMFS using cvmfsexec (list of repos can come from an environment variable or command-line argument, use same steps as above)
- integrate the parts above
- think of anything else useful and add it (improve output/error/debug messages, use info/warn/error/debug functions, ...)
- test on platforms in order
These are all suggestions.
Feel free to make differently.
Deliverables should all be recorded (text in this ticket, a new ticket, or attached or linked document, code in git).
The first step will be describing w/ words the steps more in detail.
I will review that and you can continue w/ the coding.
If you have already an idea and prefer to submit directly the code, feel free to do that.
#4 Updated by Namratha Urs 10 months ago
- % Done changed from 0 to 20
- Status changed from New to Work in progress
Prototype: Prepare a script that automates system checks and performs required steps to mount CVMFS on a worker node environment.
Script 1:
Perform required system checks (OS variant, kernel version, unprivileged userns, and others) and store the results in variables for later use OS, OS_VARIANT, KERNEL_NUM, KERNEL_PATCH_NUM CVMFS_MNT, IS_CVMFS_MNT UNPRIV_USERNS_1, IS_UNPRIV_USERNS_1 UNPRIV_USERNS_2, IS_UNPRIV_USERNS_2 FUSERMOUNT, IS_FUSERMOUNT FUSE_INSTALLED, IS_FUSE_INSTALLED USR_IN_FUSE_GRP, IS_USR_IN_FUSE_GRP Log the above variables (print to stderr) for easy debugging (collect info about the nodes that can be useful for troubleshooting and gather stats about what is out there) Check if CVMFS is mounted on the worker node using IS_CVMFS_MNT if IS_CVMFS_MNT is 0 mount required CVMFS repos with appropriate output/error messages (using mount command) else if OS = rhel if OS_VARIANT = 7.8 with KERNEL_NUM >= 3.10.0 and KERNEL_PATCH_NUM >= 1127 # check for unprivileged user namespaces if IS_UNPRIV_USERNS_1 = 0 and IS_UNPRIV_USERNS_2 = 0 # unprivileged userns is enabled use cvmfsexec to mount CVMFS repositories else # i.e. either IS_UNPRIV_USERNS_1 != 0 or IS_UNPRIV_USERNS_2 != 0 or both # unprivileged userns is not enabled use mountrepo/umountrepo to mount CVMFS repositories elif OS_VARIANT = 7.6 # check for unprivileged user namespaces if IS_UNPRIV_USERNS_1 = 0 and IS_UNPRIV_USERNS_2 = 0 if IS_FUSERMOUNT = 0 # unprivileged userns is enabled and fusermount is available use cvmfsexec to mount CVMFS repositories else # unprivileged userns is enabled and fusermount is not available ----- THINK ABOUT THIS CASE! ----- else # i.e. either IS_UNPRIV_USERNS_1 != 0 or IS_UNPRIV_USERNS_2 != 0 or both # unprivileged userns is not enabled use mountrepo/umountrepo to mount CVMFS repositories elif OS_VARIANT >= 6 and OS_VARIANT < 7 if IS_UNPRIV_USERNS_1 != 0 and IS_UNPRIV_USERNS_2 != 0 # no unprivileged userns if IS_FUSE_INSTALLED = 0 and IS_USR_IN_FUSE_GRP = 0 # FUSE rpm is installed and the user is in 'fuse' group use mountrepo/umountrepo to mount CVMFS repositories else # either FUSE is not installed or user not in 'fuse' group or both ----- THINK ABOUT THIS CASE! ----- elif OS_VARIANT = 8 and KERNEL_NUM >= 4.18 # unprivileged user namespaces are enabled (by default) use cvmfsexec to mount CVMFS repositories else # OS != rhel (any non-rhel OS) ----- THINK ABOUT THIS CASE! -----
Note that the prototype includes a high-level design to start with and will undergo incremental changes (more descriptions about the low-level details when it comes to the process of mounting the CVMFS repositories, use of exec command, scenarios which need further thought etc.) throughout the development phase. Changes to the prototype design will be periodically updated in this thread to improve the prototype over time.
#5 Updated by Namratha Urs 9 months ago
- Start date changed from 06/18/2020 to 07/06/2020
- Due date set to 07/06/2020
due to changes in a related task: #24634
#6 Updated by Namratha Urs 9 months ago
- Start date changed from 07/06/2020 to 06/29/2020
- Due date set to 07/06/2020
due to changes in a related task: #24635
#7 Updated by Namratha Urs 9 months ago
- Due date set to 07/24/2020
due to changes in a related task: #24636
#8 Updated by Namratha Urs 9 months ago
- Due date changed from 07/24/2020 to 07/17/2020
due to changes in a related task: #24636
#9 Updated by Namratha Urs 9 months ago
- Due date set to 07/17/2020
due to changes in a related task: #24637
#10 Updated by Dennis Box 5 months ago
- Target version changed from v3_7_1 to v3_7_2
#11 Updated by Marco Mambelli 4 months ago
- Target version changed from v3_7_2 to v3_7_3
#12 Updated by Marco Mambelli 3 days ago
- Target version changed from v3_7_3 to v3_7_4