Project

General

Profile

Feature #21711

Add a portable condor_chirp for jobs running under GlideinWMS

Added by Marco Mambelli about 1 year ago. Updated about 1 month ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
-
Target version:
Start date:
01/16/2019
Due date:
% Done:

100%

Estimated time:
Stakeholders:
Duration:

Description

GlideinWMS does include condor_chirp
As requested by CMS [#6424], [#6430] to allow jobs to use CHIRP.
With the introduction of Singularity condor_chirp is not always available:
- the new environment may hide the Glidein condor to the job
- the job may run on a different platform (e.g. Glidein uses base OS, EL7, and job asks for and runs in EL6)

Some VOs ran into errors while trying to use the Glidein provided condor_chirp

A solution could be to download and make available a basic condor for the platform of the Singularity image, or make always available binaries for a set of platform or provide a multy-platform client (e.g. the python client https://github.com/htcondor/htchirp )


Related issues

Related to GlideinWMS - Milestone #19515: Roadmap for Singularity supportNew03/27/2018

History

#1 Updated by Marco Mambelli about 1 year ago

As from an email from Jaime:

ssh-to-job and chirp should work with singularity jobs. For chirp, the singularity image will need to include a chirp client. There are several options for this:
  + The condor_chirp tool, which depends on the condor and classad libraries.
  + A pure python module, available here: https://github.com/htcondor/htchirp
  + Write your own client, as documented here: http://ccl.cse.nd.edu/software/chirp/

#2 Updated by Marco Mambelli about 1 year ago

#3 Updated by Marco Mambelli 8 months ago

  • Target version changed from v3_5_1 to v3_6_1

#4 Updated by Marco Mambelli 7 months ago

  • Target version changed from v3_6_1 to v3_6_2

#5 Updated by Marco Mambelli 5 months ago

  • Assignee set to Bruno Coimbra

The htchirp library is just an API that allows to use chirp functionalities in Python.
Users use condor_chirp, a client program.
This ticket should produce a drop-in replacement of condor_chirp, https://research.cs.wisc.edu/htcondor/manual/v8.7.9/Condorchirp.html
e.g. condor_chirp.py - accepting the same command-line options and behaving similarly

This should be distributed in GWMS together w/ the needed requirements (htchirp)
Possibly should be compatible w/ both python 2 (2.6, 2.7) and 3 (3.6).

http://research.cs.wisc.edu/htcondor/manual/current/condor_chirp.html
https://github.com/htcondor/htchirp

Note that the ND chirp server does not support HTCondor chirp default authentication. Tests should be done w/in a condor (interactive) job.
The goal once this is done is to package it got the Glideins and to contribute the code to the HTCondor team.
GlideinWMS will not maintain this code.

I started a github repo, pychirp, so we can test using the new development model. Please fork the repo and go from there: https://github.com/glideinWMS/pychirp

#6 Updated by Bruno Coimbra 2 months ago

  • % Done changed from 0 to 100
  • Status changed from New to Feedback

A new Python-based condor_chirp was added as a drop-in replacement for the original condor_chirp.
These changes also include gwms-python, a wrapper script for Python to be used by GlideinWMS scripts written in this language.
The new condor_chirp uses htchirp client which we are shipping as a tarball while the newest version of the package is not published.

The changes can be found on the following branch:
v36/21711-condor_chirp

#7 Updated by Marco Mambelli 2 months ago

  • Assignee changed from Bruno Coimbra to Marco Mambelli

#8 Updated by Marco Mambelli 2 months ago

  • Assignee changed from Marco Mambelli to Bruno Coimbra

#9 Updated by Marco Mambelli about 1 month ago

  • Status changed from Feedback to Resolved


Also available in: Atom PDF