Project

General

Profile

Feature #24405

Migrate to Python3: working package

Added by Marco Mambelli 7 months ago. Updated 14 days ago.

Status:
Closed
Priority:
Normal
Category:
-
Target version:
Start date:
07/07/2020
Due date:
% Done:

100%

Estimated time:
(Total: 0.00 h)
Stakeholders:
Duration:

Description

Python3 migration
  • test v3.6 vs v3.7 fe/fac fac/fe
  • Provide starting branch (Marco)
  • Re-apply futurize phase 2 or 2to3
  • compare w/ previous version migrated, especially manual changes
  • unit/integration tests in python3 (py3 only 3.6,3.8)
  • Packaging will be similar to the current GlideinMonitor one
    • Add Python package (setup tools)
    • Working RPM package
  • Smoke tests, more in-depth tests and comparisons
  • Release v3_p3
Marco, I am currently testing the first python 3.6 cut of the decision engine and modules 
but realize that it needs compatible python 3.6 rpms for the glideinwms libraries.
It looks like the libraries themselves can run on either but they have some python2 dependencies.
Do you have a clean python3 set of the glideinwms frontend library--the equivalent 
of the python in glideinwms-vofrontend-standalone and glideinwms-libs?
AND
It will not be possible to fully test the python3.6 modules in the new release without it.
Also seems that the dependencies of the rpms as they currently exist are a bit of overkill.
Is it really necessary to have the whole osg-wn-client installed on a frontend?
Also several of the rpms explicitly depend on python2 sub packages, for example python-condor and python-rrdtool.

Steve

Subtasks

Feature #24607: Fix or replace the configuration persistency in creation/lib/*DictFile.pyClosedBruno Coimbra

Feature #24608: Run the python3 pylintClosedMarco Mambelli

Feature #24750: Replace custom dictionaries in existing codeClosedMarco Mambelli


Related issues

Related to GlideinWMS - Feature #20227: Migrate to Python3New06/27/2018

Blocked by GlideinWMS - Bug #24650: Initial fix unit tests in python3Closed07/22/2020

History

#1 Updated by Marco Mambelli 7 months ago

  • Target version changed from v3_7_x to 2122
  • Description updated (diff)

#2 Updated by Marco Mambelli 7 months ago

#3 Updated by Marco Mambelli 6 months ago

  • Assignee set to Thomas Hein

Here a reminder on how to Install GlideinWMS framework on Fermicloud using OSG RPMs.

1. You will need one host for the Factory (including the condor Factory collector and schedd) and one for the Frontend (including the condor User collector/negotioator and a schedd).
Take note of the 2 host addresses (hostname) you will need them for the configuration
Take note also of the certificates DNs. Certificates are in /etc/grid-security/host*

To make new VMs and administer them use fcluigpvm01:
ssh fcluigpvm01.fnal.gov

This script will list your hosts (ID and hostname needed to ssh):
~marcom/bin/myhosts

Then login as root on your VMs:
ssh
And follow the install instructions

2. The factory installation instructions are in the OSG documentation. Follow them to install it:
https://opensciencegrid.org/operations/services/install-gwms-factory/

3. The frontend installation instructions are in the OSG documentation. Follow them to install it :
https://opensciencegrid.github.io/docs/other/install-gwms-frontend/

Continue as much as you can and provide also feedback about the instructions

v3.6.2 is in OSG production (osg repo)
v3.7 is in OSG upcoming (osg-upcoming repo)

#4 Updated by Marco Mambelli 6 months ago

Use Git in the install

Factory

exclude glidein packages from updates:
vi /etc/sysconfig/yum-autoupdate
EXCLUDE="kernel* openafs* *-kmdl-* kmod-* *firmware* condor* glidein*" 

# In SL7
vi /etc/yum/yum-cron.conf
# Do not update these automatically
exclude=kernel* openafs* *-kmdl-* kmod-* *firmware* condor* glidein*

# sl7 does not have this file. Alt. add at the end of yum.conf:
vi /etc/yum.conf
## Exclude following Packages Updates ##
exclude=glidein* condor*

yum install -y git
mkdir /opt/gwms-git
cd /opt/gwms-git
git clone ssh://p-glideinwms@cdcvs.fnal.gov/cvs/projects/glideinwms
cd glideinwms
# you may need a kinit if the key was not forwarded
git checkout v3/6983

# For SL6
export MYPYTHON=python2.6
# For SL7
export MYPYTHON=python2.7

pushd /usr/sbin
mkdir -p /opt/fromrpm/usr_sbin

mv checkFactory.py* glideFactoryEntryGroup.py* glideFactoryEntry.py* glideFactory.py* manageFactoryDowntimes.py* stopFactory.py* /opt/fromrpm/usr_sbin/
for i in checkFactory.py glideFactoryEntryGroup.py glideFactoryEntry.py glideFactory.py manageFactoryDowntimes.py stopFactory.py; do ln -s /usr/lib/$MYPYTHON/site-packages/glideinwms/factory/$i $i; ln -s /usr/lib/$MYPYTHON/site-packages/glideinwms/factory/${i}o ${i}o; ln -s /usr/lib/$MYPYTHON/site-packages/glideinwms/factory/${i}c ${i}c; done

# already in: pushd /usr/sbin
mkdir -p /opt/fromrpm/usr_sbin
mv clone_glidein glidecondor_createSecCol glidecondor_addDN glidecondor_createSecSched info_glidein reconfig_glidein /opt/fromrpm/usr_sbin
## Not in factory  mv reconfig_frontend /opt/fromrpm/usr_sbin   # reconfig_frontend is not there at start
for i in glidecondor_createSecCol glidecondor_addDN glidecondor_createSecSched; do ln -s /opt/gwms-git/glideinwms/install/$i $i; done
for i in clone_glidein info_glidein reconfig_glidein reconfig_frontend; do ln -s /opt/gwms-git/glideinwms/creation/$i $i; done

mkdir -p /opt/fromrpm/usr_bin
mv /usr/bin/glidein* /opt/fromrpm/usr_bin/
pushd /usr/bin/
for i in glidein_cat glidein_gdb glidein_interactive glidein_ls glidein_ps glidein_status glidein_top; do  ln -s /usr/lib/$MYPYTHON/site-packages/glideinwms/tools/${i}.py $i; done

cd /usr/lib/$MYPYTHON/site-packages/
mv glideinwms/ fromrpm_glideinwms
mkdir glideinwms
cd glideinwms
for i in creation  factory  __init__.py lib  tools; do ln -s /opt/gwms-git/glideinwms/$i $i; done

# no touching files in /etc

cd /var/lib/gwms-factory/
mkdir fromrpm
mv creation web-base fromrpm/
ln -s /opt/gwms-git/glideinwms/creation creation
ln -s /opt/gwms-git/glideinwms/creation/web_base web-base

cd /var/log/gwms-factory/
mkdir old
mv client server old
mkdir client server
chown gfactory: server

# For SL6
/etc/init.d/gwms-factory upgrade

# For SL7
/usr/sbin/gwms-factory upgrade
#/usr/sbin/gwms-factory reconfig
/bin/systemctl start gwms-factory

Frontend

sw from git repository testing install
# exclude glidein packages from updates:
vi /etc/sysconfig/yum-autoupdate
EXCLUDE="kernel* openafs* *-kmdl-* kmod-* *firmware* condor* glidein*" 

# sl7 does not have this file. Alt. add at the end of yum.conf:
vi /etc/yum.conf
## Exclude following Packages Updates ##
exclude=glidein* condor*

yum install -y git
mkdir /opt/gwms-git
cd /opt/gwms-git
git clone ssh://p-glideinwms@cdcvs.fnal.gov/cvs/projects/glideinwms
# you may need a kinit if the key was not forwarded
cd glideinwms
git checkout v3/6983

# For RHEL 6
export PYVERSION=2.6

# For RHEL 7
export PYVERSION=2.7

mkdir -p /opt/fromrpm/usr_sbin
cd /usr/sbin
mv checkFrontend glideinFrontendElement.py glideinFrontend stopFrontend /opt/fromrpm/usr_sbin/
for i in checkFrontend glideinFrontend stopFrontend; do ln -s /usr/lib/python$PYVERSION/site-packages/glideinwms/frontend/${i}.py $i; done
ln -s /usr/lib/python$PYVERSION/site-packages/glideinwms/frontend/glideinFrontendElement.py glideinFrontendElement.py

# already in /usr/sbin
mv glidecondor_addDN glidecondor_createSecCol glidecondor_createSecSched /opt/fromrpm/usr_sbin/
for i in glidecondor_addDN glidecondor_createSecCol glidecondor_createSecSched ; do ln -s /opt/gwms-git/glideinwms/install/$i $i ; done

# already in /usr/sbin
mv reconfig_frontend /opt/fromrpm/usr_sbin/
ln -s /opt/gwms-git/glideinwms/creation/reconfig_frontend reconfig_frontend

mkdir -p /opt/fromrpm/usr_bin
cd /usr/bin/
mv /usr/bin/glidein* /opt/fromrpm/usr_bin/
for i in glidein_cat  glidein_gdb  glidein_interactive  glidein_ls  glidein_ps  glidein_status  glidein_top; do  ln -s /usr/lib/python$PYVERSION/site-packages/glideinwms/tools/${i}.py $i; done

cd /usr/lib/python$PYVERSION/site-packages/
mv glideinwms fromrpm_glideinwms
mkdir glideinwms
cd glideinwms/
for i in creation  frontend  __init__.py  lib  tools; do  ln -s /opt/gwms-git/glideinwms/$i $i; done
pushd /opt/gwms-git/glideinwms/creation/
ln -s . creation
popd

# no touching files in /etc

cd /var/lib/gwms-frontend/
mkdir fromrpm
mv creation web-base fromrpm/
ln -s /opt/gwms-git/glideinwms/creation creation
ln -s /opt/gwms-git/glideinwms/creation/web_base web-base

vi /etc/gwms-frontend/frontend.xml

service gwms-frontend upgrade
service gwms-frontend reconfig
service gwms-frontend start

#5 Updated by Marco Mambelli 5 months ago

Status as of 7/7

Working branch is branch_v3_p3_a2
Testing machines are fermicloud018.fnal.gov and fermicloud020.fnal.gov
The goal is to have a fully python3 version w/o six or other compatibility libraries.

All automatic filters were applied.
The deprecated string functions were replaced w/ the methods.
The classes persisting the configuration dictionaries (creation/lib/) are causing problems:
Changed DictFile to use binary files (bytes) and convert the strings in the dictionaries
Since DictFiles values may include text but also the content of binary files, they must be treated as bytes. Dictionaries and keys though are regular strings, so the interface is set when writing to the file or the in-memory stream (io.BytesIO). The current version seems not to crash (no stack trace) but the reconfig on the Factory does not complete in a reasonable time.
Further work is needed, maybe in a sub-ticket.

Thomas is working on the pylint script for python3 (do_pylint.sh)

The release branch may remain branch_v3_p3_a2, progress should be done in ticket branches names v3p3/TICKET_NR

#6 Updated by Marco Mambelli 5 months ago

  • Start date changed from 05/13/2020 to 07/07/2020
  • Due date changed from 06/01/2020 to 07/07/2020

due to changes in a related task: #24607

#7 Updated by Marco Mambelli 5 months ago

  • Due date set to 07/07/2020

due to changes in a related task: #24608

#9 Updated by Marco Mambelli 4 months ago

  • Assignee changed from Thomas Hein to Marco Mambelli

#10 Updated by Marco Mambelli 4 months ago

  • Blocked by Bug #24650: Initial fix unit tests in python3 added

#11 Updated by Marco Mambelli 4 months ago

In some notes:

2to3 -W -n -f fixname

#12 Updated by Marco Mambelli 4 months ago

  • Due date set to 08/11/2020

due to changes in a related task: #24750

#13 Updated by Marco Mambelli 2 months ago

  • Target version changed from 2122 to v3_9_1

#14 Updated by Marco Mambelli 15 days ago

  • Target version changed from v3_9_1 to v3_9_0
  • Status changed from New to Resolved

#15 Updated by Marco Mambelli 14 days ago

  • Status changed from Resolved to Closed

Also available in: Atom PDF