Project

General

Profile

GlideinWMS roadmap and HEPCloud

10/15/2019
wh12se - long baseline
https://fnal.zoom.us/j/695119889

Attending: Krista, Tony, Kevin, Marco, Burt

Tony answered to the questions that Marco provided

  1. Will GWMS be involved in discussing and defining the interactions with HPC facilities (e.g. ALCF) or will HEPCloud provide requirements with the desired GWMS features?
    HEPCloud will provide the requirements.
    GlideinWMS will not be involved in contacts and working relationships discussions that have been already started.
    HEPCloud will do the groundwork and GlideinWMS should wait and will start the work once the requirements and modalities to use HPC have been defined.
  2. Should GWMS improve its capability to submit multi-node jobs?
    yes. GlideinWMS should improve internal accounting for multi-node jobs.
    There are problems on how pressure is being handled at TACC and NERSC for muti-node jobs, ask Steve for a more precise description
  3. Should GWMS support different resource provisioning models?
    NO GlideinWMS is optimized for the Grid.
    This would become a distraction from that work.
    HEPCloud has a different provisioning model
    There are a couple of R&D projects in HEPCloud
  4. Will HEPCloud replace GWMS or part of it?
    We are not replacing GlideinWMS
    HEPCloud will continue to use GlideinWMS for Grid provisioning.
    To work better with HEPCloud GlideinWMS should consider moving to the Frontend to use a DE Channel model
    The DE is a framework. HEPCloud would like a Frontend library that could be executed through the DE channel
    GlideinWMS as a project may want to be involved in making the Frontend less monolithic and adopt the HEPCloud Decision Engine framework (data blocks - dictionaries passed around)

Marco understood a more active role of GlideinWMS from previous discussions, especially about 1 and 3, but this is the current HEPCloud plan.

Digression describing HEPCloud
Has components:
  • sources
  • outputs
  • transforms
  • logic engine (filter)

Decision Engine is a channel
HEPCloud is a framework, has also other envisioned components like the prediction engine
Has a rule engine based on Nova logging subsystem - facts (true, false, rules)
https://hepcloud.fnal.gov/
https://hepcloud.fnal.gov/architecture/
https://cd-docdb.fnal.gov/cgi-bin/sso/RetrieveFile?docid=5838&filename=FNALHEPCloudFacility.pdf&version=2

CMS and OSG are asking for updates on the Decision Engine and when they can put their hands on it.
  • Provide a set of decision channels
  • Use the DE framework API
  • HEPCloud project would define the DE Framework
  • GlideinWMS should consider moving the Frontend to that framework