Project

General

Profile

Support #24897

Run and build Minos code under SL6 singuilarity

Added by Arthur Kreymer 2 months ago. Updated 26 days ago.

Status:
Work in progress
Priority:
High
Start date:
09/01/2020
Due date:
11/01/2020
% Done:

30%

Estimated time:
100.00 h
Duration: 62

Description

Minos code presently builds and runs under SLF6, with is EOL Nov 2020.
For data preservation, and to allow short term running,
we plan to build and run Minos code in a Singularity container.

History

#1 Updated by Arthur Kreymer 2 months ago

I found a very useful document by Marco Mambelli, giving clean examples of usage.
https://indico.fnal.gov/event/20902/contributions/59849/attachments/37476/45542/FIFE_Singularity-190530.pdf

I tested this on anniesl7gpvm01, where singularity is installed.

SIN=/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl6:latest

singularity shell ${SIN}

Singularity> pwd
/nashome/k/kreymer

Singularity> cat /etc/redhat-release
Scientific Linux release 6.10 (Carbon)

Singularity is not installed on Minos nodes.
I have issued RITM1014283 09/01 Singularity on Minos SL7 systems

#2 Updated by Arthur Kreymer about 2 months ago

  • % Done changed from 0 to 10

singularity has been installed on minos-data and minosttestgpvm01.

Please give Minos code a try !

I have tested singularity as follows :

MINOSDATAGPVM01 > SIN=/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl6:latest

MINOSDATAGPVM01 > singularity shell ${SIN}
Singularity> pwd
/nashome/k/kreymer

Singularity> cat /etc/redhat-release
Scientific Linux release 6.10 (Carbon)

MINOSTESTGPVM01 > SIN=/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl6:latest

MINOSTESTGPVM01 > singularity shell ${SIN}

Singularity> pwd
/nashome/k/kreymer

Singularity> cat /etc/redhat-release
Scientific Linux release 6.10 (Carbon)

#3 Updated by Arthur Kreymer about 2 months ago

We need to bridge mounted file systems, including
/minos/app
/minos/data
/grid/fermiapp
/pnfs/minos.

Something like
singularity shell -B /minos/data -B /grid/fermiapp ${SIN}
etc

I will investigate the syntax.

#4 Updated by Arthur Kreymer about 2 months ago

  • % Done changed from 10 to 20
  • Status changed from Assigned to Work in progress

I located the overall Singularity documents at

https://sylabs.io/docs/

We can get an SL6 shell session on the Minos SL7 system by doing

SIN=/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl6:latest
SFI=/minos,/grid,/pnfs
singularity shell -B ${SFI} ${SIN}

This gets you a bare shell, without your usual .bashrc or .profile
In my case, I need to do . ./.profile
Most people will want to do . ./.bashrc

#5 Updated by Arthur Kreymer about 1 month ago

  • % Done changed from 20 to 30

I created a wrapper script for getting into singularity.
If it is useful, I can move it into /minos/app/scripts

cat ~kreymer/sing

#!/bin/bash

SIN=/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl6:latest

SFI=/minos,/grid,/pnfs,/cvmfs

singularity shell -B ${SFI} ${SIN}

I am able to run a loon executeable on minos-data as follows :

. /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
 setup minos
 setup_minos -r R3.11
 loon 
 .exit

The minos cvmfs files have a problem with the datagram client configuration
which reports setups to our logging system.
datagram_client.py has SERVER_ADDRESS set to minos-nearline.
This should be minos-data.

But loon still works.

Yesterday the setup of ups failed under singularity.
Today it is working.
Today was a monthly maintenance day, perhaps something was fixed.

#6 Updated by Arthur Kreymer about 1 month ago

I have created /minos/app/scripts/sing to start up a singularity shell.
It appears the loon jobs can be set up and run on SL7 using this.

Note that the scheduled upgrade of minos61-63 did not happen 09/16.
This will be rescheduled.

For now, use minostestgpvm01 or minosdatagpvm01 for testing.

#7 Updated by Arthur Kreymer about 1 month ago

According to the document https://cdcvs.fnal.gov/redmine/projects/fife/wiki/Singularity_jobs
it seems we need to start submitting jobs with

-l '+SingularityImage=\"/cvmfs/singularity.opensciencegrid.org/fermilab/fnal-wn-sl6:latest\"'
--append_condor_requirements='(TARGET.HAS_SINGULARITY=?=true)'

instead of

--OS=SL6

#8 Updated by Arthur Kreymer about 1 month ago

Report from Anna via email :

Holin, Anna via liveuclac.onmicrosoft.com
4:49 AM (5 hours ago)

to me, Tom, Jennifer, Karol

Hello Art,

as promised, I tried to do the setting up of a test release in the singularity. So what I did was go into the slf7 test machine, then I setup minos R3.11; so far so good, loon works too.
I then set up a test release - it seems to have worked just fine. I gave it the rather unfortunate name of
/minos/app/annah1/testslf7

(given that it's actually supposed to be an slf6 singularity). Anyway, I then tried to add a package. This is when I got an error, see all my output below. Maybe it's easy to solve, seems like it's some sort of key forwarding problem in the singularity, but it looks like currently I can create a test release in the singularity, but not add packages. Please let me know what you think...
Many thanks in advance,

Anna

<minostestgpvm01.fnal.gov> ./sing
Singularity> . .profile
<minostestgpvm01.fnal.gov> . /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
<minostestgpvm01.fnal.gov> setup_minos -r R3.11
bash: /afs/fnal.gov/files/code/e875/general/condor/scripts/setup_minos_condor.sh: No such file or directory
setup_minossoft_FNALU forcing 64bit build
setup_minossoft_FNALU_64bit_parser forcing flavor: Linux64bit+2.6-2.12
srt_setup SRT_BASE_RELEASE=R3.11 SRT_CXX=GCC_4_9 SRT_QUAL=e9-r5-debug
 /grid/fermiapp/minos/minossoft_srt_64bit/releases/boot/bin/generic/srt_environment
  used to hack SRT_ARCH=Linux3.1 to Linux2.6 
MINOSSOFT release "R3.11" 
SRT_SUBDIR=Linux2.6-GCC_4_9-e9-r5-debug
ROOT=v5_34_32 Linux64bit+2.6-2.12-e9-nu-debug
CONFIG=v02
adjust_tsql.sh reader left alone EXPERIMENT env ():GRID_GROUP env ():GROUP env ():id -g -n (e875)
<minostestgpvm01.fnal.gov> 
<minostestgpvm01.fnal.gov> cd /minos/app/annah1/
<minostestgpvm01.fnal.gov> newrel -t R3.11 testslf7 
Creating a test release "testslf7" in the directory
    /minos/app/annah1
<minostestgpvm01.fnal.gov> cd testslf7/
<minostestgpvm01.fnal.gov> 
<minostestgpvm01.fnal.gov> srt_setup -a
 /grid/fermiapp/minos/minossoft_srt_64bit/releases/R3.11/bin/Linux2.6-GCC_4_9-e9-r5-debug/srt_environment
  used to hack SRT_ARCH=Linux3.1 to Linux2.6 
<minostestgpvm01.fnal.gov> addpkg -h NuMuBar
Adding package "NuMuBar" to ".".
cvs checkout -P NuMuBar
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@       WARNING: POSSIBLE DNS SPOOFING DETECTED!          @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
The RSA host key for minoscvs.fnal.gov has changed,
and the key for the corresponding IP address 131.225.108.132
is unknown. This could either mean that
DNS SPOOFING is happening or the IP address for the host
and its host key have changed at the same time.
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
01:30:15:c1:ab:4a:41:c3:62:e2:46:35:71:2e:c7:64.
Please contact your system administrator.
Add correct host key in /nashome/a/annah1/.ssh/known_hosts to get rid of this message.
Offending key in /nashome/a/annah1/.ssh/known_hosts:2
RSA host key for minoscvs.fnal.gov has changed and you have requested strict checking.
Host key verification failed.
cvs [checkout aborted]: end of file from server (consult above messages if any)
cvs checkout failed.
addpkg failed.
<minostestgpvm01.fnal.gov>

#9 Updated by Arthur Kreymer about 1 month ago

This was tricky !

The Singularity image does not have a functional /etc/krb5.conf file,
so you cannot kinit or ssh from this environment.
Hence no access to CVS.

I have created a workaround :
We need to set an environment that uses a current krb5.conf,
and that does ssh with the appropriate default options.

export KRB5_CONFIG=/minos/app/scripts/krb5.conf
export CVS_RSH=/minos/app/scripts/cvsrsh

This works for me, please give it a try

#10 Updated by Arthur Kreymer about 1 month ago

Holin, Anna via liveuclac.onmicrosoft.com
11:51 AM (1 hour ago)
to me, Tom, Jennifer, Karol

Hello Art,

thank you for looking into this and doing the fix.
With this fix in place, I was able to add packages, run the proxy resolver, compile the code and run a test job, and it all worked just fine.
So I think this is good news!
Many thanks again!

Anna

#12 Updated by Arthur Kreymer 26 days ago

I added /opt to the mounted file systems in the /minos/app/scripts/sing script, so that shared accounts can see their configuration files under /opt.

Also available in: Atom PDF