Project

General

Profile

Feature #20227

Migrate to Python3

Added by Marco Mambelli almost 2 years ago. Updated 15 days ago.

Status:
New
Priority:
Normal
Category:
-
Target version:
Start date:
06/27/2018
Due date:
% Done:

0%

Estimated time:
(Total: 0.00 h)
Stakeholders:

HEPCloud

Duration:

Description

Python.org ( https://devguide.python.org/#status-of-python-branches ) and several python packages will drop support for Python 2.7 before 2020: http://python3statement.org/
RHEL 8 will ship with only Python3
EPEL is also pushing to port everything to Python3, which is the currently recommended python version (also in EL7): https://fedoraproject.org/wiki/SIGs/Python
RH will support Python 2.7 until RHEL7 EOL

The recommendation, if stakeholders agree, is to plan a full migration to Python 3 by summer 2019


Subtasks

Feature #20232: Run unit tests in Python 3AssignedDennis Box


Related issues

Related to GlideinWMS - Idea #24351: Revise the project directory structureNew04/27/2020

Related to GlideinWMS - Feature #18301: Apply code modernizationClosed11/15/2017

Related to GlideinWMS - Idea #23144: Start introducing Python type annotationNew08/20/2019

Related to GlideinWMS - Feature #24405: Migrate to Python3: working packageNew05/13/202006/01/2020

History

#1 Updated by Marco Mambelli almost 2 years ago

We mainly need support for m2crypto and condor (which has Python 3 support)
Consider SL7 and SL6 support
EPEL has full Python 3 support for EL7, 3.4 and 3.6:
https://dl.fedoraproject.org/pub/epel/7/x86_64/Packages/p/
EPEL for SL6 has python34:
https://dl.fedoraproject.org/pub/epel/6/x86_64/Packages/p/

Here instructions to install w/ altinstall from the Python tarballs (viable for tests, not for production deployment):
https://danieleriksson.net/2017/02/08/how-to-install-latest-python-on-centos/

Python 3.4 could be a safe choice more compatible also w/ EL6
Python 3.6 adds coroutine support (async/wait, generators, ...), strings literals, type annotation, dicts are ordered.

EL6 support is not critical. It is in product retirements phase (Maintenance Support 2, only critical security fixes and not supported in new hardware). And it will become unsupported on November 30, 2020 (Extended Lifetime support, no security updates -> Fermilab will not allow it)

#2 Updated by Marco Mambelli almost 2 years ago

  • Target version changed from v3_5_x to v_collections

#3 Updated by Marco Mambelli almost 2 years ago

v3.7 will be parallel to the Python 2 version (v3.5 or 3.6)

Consider migrating configuration to YAML (https://www.zionandzion.com/json-vs-xml-vs-toml-vs-cson-vs-yaml/)
- tool to port XML config files
- maintain the ability to include parts (config.d)
- preserve checks on what is allowed
- revisit the whole config system:
- are temp files still necessary?
- OK to rewrite config file (to complete w/ default values, to eliminate duplicates, to sort some lists). Maybe move this to a separate tool

#4 Updated by Marco Mambelli 15 days ago

  • Stakeholders updated (diff)
  • Assignee set to Marco Mambelli

HEPCloud needs the Frontend and common libraries in Python3 in order to test and run the Python 3 version of Decision Engine

#5 Updated by Marco Mambelli 15 days ago

Factory Ops (OSG, CMS, FIFE) needs the OSG 3.4 python 2 version (v3.6.x) at least until Oct 2020 (for CREAM support).
VO may have Frontends locked on older SL version during data taking (LIGO is at SL6 as of now).
Containerization may ease the transition to python3

HTCondor releases the new python bindings for python3
As mentioned earlier, HEPCloud needs Python3

Plan:
  • complete Frontend running in parallel
  • revise the repository directory structure (Python3 will follow newer standards like GlideinMonitor)
  • After cutting at least 3.6.3 RC, branch off the v3_7 + 3.6.3, branch_v3_p3
  • check with stakeholders if the development version could be Python3?
  • Re-apply Futurize, compare w/ past branch from Thomas
    • common libraries and frontend to be migrated first
  • Revise
    1. configuration
    2. credentials

#6 Updated by Marco Mambelli 15 days ago

  • Related to Idea #24351: Revise the project directory structure added

#7 Updated by Marco Mambelli 15 days ago

#8 Updated by Marco Mambelli 15 days ago

  • Related to Idea #23144: Start introducing Python type annotation added

#9 Updated by Marco Mambelli 15 days ago

  • Related to Feature #24405: Migrate to Python3: working package added


Also available in: Atom PDF