Project

General

Profile

Roadmap

Summary

Due in about 2 months (05/13/2021)

Roadmap Summary

No issues for this version

RoadmapSummary

  • High priority topics
    • Reduce operations effort
      • Easy deployment and use of tokens
      • Automatic Factory configuration generation, via CRIC
      • Improve autoconfiguration and other automation scripts in Factories
      • Provide containers for Frontend and Factory
      • Increase automation in CI/CD
    • Improve support for tokens
      • Support for tokens in GlideinWMS
        • Use HTCondor IDtokens for Glideins authentication
        • Support submission of Glideins to sites using sci-token
        • Authenticate Factory and Frontend using HTCondor IDtokens
      • Tokens as first-class credential
      • Ease token management
      • Support transitive trust via OSG Collector
    • Improve modularity and code quality (especially of Frontend)
      • Move to Python 3
        • Fix remaining bugs with the Python 3 implementation
      • Improve code quality
      • Separate out the Glidein code
        • Extend and ease the support of custom scripts
      • Make the code more modular (especially the Frontend)
      • Expand, simplify, and automate testing
      • Keep supporting Python 2 until mid-2021 or until OSG 3.5 is supported
    • Singularity support
      • Allow VO scripts also inside Singularity
      • Allow CVMFS/Singularity when none is available on a worker node
      • Singularity invocation via HTCondor
      • Support condor_ssh_to_job for Singularity jobs
    • Use more of HTCondor features in GWMS
      • Blackhole prevention
      • Singularity invocation via HTCondor
      • Use of tokens (security without x509 certificates)
    • Upgrade Frontend and Factory monitoring
      • Collaborate with UCSD (Edgar) and merge the results in production

v3_6_7

GlideinWMS 3.6.7

62%

16 issues   (10 closed — 6 open)

Related issues
Bug #4619: Factory info logs message regarding classad advertising
Bug #5241: Frontend credential selection plugin ProxyUserMapWRecycling seems to be broken
Bug #23187: Active entries not corresponding with the ones defined in the glideinWMS.xml
Bug #25350: fix collector_setup.sh to allow hyphens in the hostname and add unit tests
Bug #25560: Define RELEASE_DIR and BIN for the GLidien HTCondor
Feature #5345: Expand attribute expansion capabilities
Feature #6556: Frontend massively overprovisions sites due to no grouping of entries
Feature #7815: Manage OOM score of frontend processes
Feature #11754: Allow controlling of site/entry's limits from the frontend
Feature #11756: Is it time to get rid of TotalTimeUnclaimedIdle?
Feature #11802: Ability to add custom information to the EC2 USER META DATA
Feature #22988: Move files to be accessed and tracked by others
Feature #24540: DO not check for pilots inside groups that are disabled
Feature #25257: Expand condor_platform_select.sh auto detection to recognize powerpc architectures
Feature #25325: Improve GitHub actions not to fail pycodestyle all the times
Feature #25436: Extend gwms-logcat.sh to extract other log files (startd slots and extra ones)

v3_6_x

Placeholder for v3.6 series (OSG 3.4 and 3.5 production) issues

No issues for this version

v3_7_3

GlideinWMS 3.7.3

11%

51 issues   (1 closed — 50 open)

Related issues
Bug #21989: Make attribute publishing independent from other properties
Bug #23052: parameter=True ignored in Factory attributes if const=True or publish=False
Bug #23339: Make wholenode knobs uniform
Bug #23376: Possible bug if there are extra spaces at the beginning or at the end of a value in glidein_config
Bug #25176: root on the Factory host unable to remove the Glidein
Bug #25224: Fix glidein_startup.sh in the staging area to be the one w/ the payload
Bug #25260: Glideinmon: validate configuration before starting up
Bug #25299: permit IDTOKENS when querying schedds
Bug #25306: Polish condor configuration
Bug #25438: Reconfig error if some variables are set to a value different form the default
Bug #25470: Fix a few issues generated when adding an http sub package
Bug #25490: tmpfile on separate device fails to rename when factory writes tokens
Bug #25504: factory should not check condor_tarball version prior to attempting SciTokens authentication to CE
Bug #25557: Improve error message when factory econfigure fails with a duplicated entry
Bug #25570: IDTOKENS doesn't work with condor 8.9.10 or later
Feature #21313: Containerize Factory deployment
Feature #21314: Containerize Frontend deployment
Feature #21558: Add the ability to advertise an attribute in the entry classad also if constant
Feature #23091: Reliable, flexible and secure logging system for distributed workflows
Feature #23136: Add an attribute to save the logging channel content in the Glidein stderr
Feature #23265: Add security mechanisms to the glidein logging channel
Feature #24465: Testing framework for distributed resources and GlideinWMS
Feature #24491: Multi Pronged Approach to Pilot Log Anonymization
Feature #24536: Consider yapf or another auto-formatter for GWMS auto-formatting
Feature #24546: Add the ability to use CVMFS on sites that do not provide local installations
Feature #24616: Improve the handling of curl and wget
Feature #24677: Add the possibility to override "attributes" (-params...) in manual_glidein_startup
Feature #25073: Allow VO scripts to run before the job (in Singularity if used)
Feature #25096: Support to run periodic scripts within Singularity
Feature #25143: Add a mechanism to download updated (VO) setup scripts
Feature #25200: Consider tokens for glidein submission the same way as grid_proxy, ssh keys, ...
Feature #25220: Change the order of places where the Glidein is looking for singularity
Feature #25225: Integrate glidein in a vacuum in the factory
Feature #25247: Improve the generation of Glideins' idtokens
Feature #25249: Review the need for GLIDEIN_REQUIRE_VOMS, VomsIdle, ProxyIdle
Feature #25252: Update unit test for OSG 3.5 and singularity
Feature #25253: Update (monitoring) tools to support Singularity
Feature #25282: Make the the Frontend more resilient to slow/overloaded HTCondor frontends
Feature #25428: Remove LD_PRELOAD from the environment by default
Feature #25440: Fail, no fall-back when the user-provided singularity image is missing
Feature #25450: Change IDTOKEN use to be more aggressive by delaying the checks
Feature #25464: Cleaned up GPU implementation removing host lib
Feature #25477: Add layer to account for pilot entries in OSG_autoconf
Feature #25510: Allow also a PATH with multiple directories, not only a single file path, to override the singularity binary
Idea #24635: Prepare separate scripts for mount and unmount of CVMFS

v3_7_4

GlideinWMS 3.7.4

2%

83 issues   (0 closed — 83 open)

Related issues
Bug #5414: Multiple proxies not being handled correctly - numbers too low
Bug #7243: Frontend support for HA Collectors
Bug #15943: Overwriting of /etc/gwms-factory/glideinWMS.xml
Bug #16702: Fix scripts install/glidecondor_createSecCol and install/glidecondor_createSecSched
Bug #20861: Error message related to entry in the Factory logs
Bug #21832: PicklingError: Can't pickle <type 'Boost.Python.enum'>: import of module Boost.Python failed
Bug #23038: Directory /var/lib/gwms-frontend/vofrontend/monitor blocks upgrade and reconfig commands
Bug #23089: fermifactory02 monitoring is wrong somehow
Bug #23176: unit test failures after changing to python htcondor bindings
Bug #23768: Bug in multiple credential distribution
Bug #24257: glideFactory.py should not create keys and url lists if remote logging not enabled
Bug #24399: Pilot draining: add the possibility of specifying the downtime lenght
Bug #24402: Not all the logfiles are rotated in the factory
Bug #24516: Confusing verification script
Bug #24610: Glidein pressure too low for HPC systems in HEPClod
Bug #24657: Document monitor xml files and produce equivalent json
Bug #24658: Factory is not respecting global limits when multiple VOs are present
Bug #24746: Empty variables or with only spaces are not set in the job environment
Bug #24794: Fix the output messages in the CI scripts
Bug #24806: Missing documentation for GLIDEIN_IDLE_LIFETIME
Bug #25150: Inconsistency in the handling of parameters/attributes
Bug #25351: Possible bug if dash is a valid character in HTCondor sock names
Feature #2121: VO Frontend missing features for Cloud
Feature #2531: Monitoring for frontend: store the number of Job restarts
Feature #5652: Add support for flocked collectors
Feature #10397: Identify means to pass some job related information to the glideins requested
Feature #11500: Support condor_q -autocluster to increase performance
Feature #13416: Propagate attributes from the job classad to the glidein JDL
Feature #13854: Migrate to signing type to sha256
Feature #13965: Set up a project repo with the glideinwms rpms
Feature #15280: Support auto for memory just like we do for cores
Feature #18148: Enable proxy extension for Glideins
Feature #18855: Add unit test for lib/fork.py
Feature #19214: Add a configurable limit to the rate of jobs running and fail the glidein if the rate is passed
Feature #19947: Do not restart condor on factory upgrade
Feature #19948: Clean up configuration files from disabled entries
Feature #19949: Remove really old files from reconfig.
Feature #20237: Design a logo for GlideinWMS
Feature #20505: Consider if the blacklisting script from FIFE can be added as GlideinWMS frontend feature
Feature #20776: Ask for feedback about the new Singularity scripts and remove the TODOs
Feature #20811: Adopt Singularity mechanisms provided by HTCondor
Feature #21292: Add GCE termination command as suggested by Burt
Feature #21741: Improve monitoring stats in glidefactory and glidefactoryclient classads
Feature #21875: Invoke Singularity via HTCondor INSTEAD of GWMS job wrapper
Feature #21886: Custom modules for job wrappers
Feature #22036: Dots seem to cause problems in entry names
Feature #22163: Check if there are load changes in Factory and solve TODOs added in #21880
Feature #22190: Remove client blacklist from the factory
Feature #22284: Allow the Singularity wrapper to ensure unicity of bind mounts
Feature #22846: Increase use of HTCondor Python bindings and add them as requirement
Feature #22848: Improve Glidein monitoring and troubleshooting
Feature #22866: Create dynamic pages serving the Glideins stdout, stderr and included content
Feature #23053: Add attributes that are VO and site dependent
Feature #23253: Blackhole detection logs: Publishing expression/classAd attributes to the StartdLogs
Feature #23290: Support condor_ssh_to_jpb also for Singularity jobs
Feature #23486: Make Glidein's condor resilient to missing or mis-configured hostname
Feature #23578: Review Glidein variables: lifetime, naming, defaults (especially the ones used in Singularity)
Feature #23906: Create a script to prune and purge the web-area
Feature #24139: Add LOG_RECIPIENTS_CLIENT to support log servers specification on the Frontend
Feature #24306: Evaluate non exec-ing singularity in the wrapper to be able to examine the execution and report better errors
Feature #24407: Support CVMFS and Singularity when none of the 2 is available on the worker node
Feature #24440: Contact directly the schedd for queries using python bindings
Feature #24441: Increase the number of Gridmanagers in the Factory
Feature #24553: Update "pychirp"
Feature #24554: Warn about grid resource unavailable
Feature #25001: Conside muving the Factory collector under shared_port
Feature #25199: Avoid crashing the Factory (and the Frontend) if condor is down/not responsive
Feature #25201: Make sure that Factory is not removing Glideins if not asked to do so
Feature #25222: Add in the CI test table links to the test logs
Feature #25304: Include test_os.sh in GlideinWMS
Feature #25334: Create in GitHub actions a report similar to Fermilab CI
Feature #25336: Greate a GitHub action to publish the GWMS API (code documentation)
Feature #25337: Create a Docker image to run locally GitHub actions
Idea #23451: Allow a Frontend to run in parallel w/o affecting the Factory
Idea #24569: Some requests/suggestions form HEPCloud

v3_7_x

Placeholder for v3.7 development series (OSG 3.5 upcoming) issues

8%

71 issues   (0 closed — 71 open)

Related issues
Bug #2169: factoryEntryStatusNow.html buggy - mixing 3 xml files
Bug #4050: Factory should handle os.fork() failing
Bug #4423: factoryCompletedStats doesn't account for multicore
Bug #4536: Problem advertising requests affecting FE logs
Bug #4587: OSG: Automate / fix HTCondor-CE issues
Bug #4589: Init script "restart" action incorrect
Bug #4617: Race in TERM handling in glideinFrontendElement
Bug #4678: Glidein reports negative retire times
Bug #4781: Secondary Condor Collectors share address files
Bug #4813: Incorrect/Misleading sample ini files in the docs
Bug #4899: Cloud environment doesn't source profile.d
Bug #6991: Document important glidefactory/client/factoryclient classad attributes
Bug #7848: If num_parallel_workers are 1 and a child crashes, factory can get stuck
Bug #11142: Collision when 2 credentials use the same file name
Bug #14869: Hardening available cores auto detection
Bug #23612: Make sure that the classad attribute names are treated as case insensitive
Feature #2408: Frontend to detect if its mapped name is not correct
Feature #2897: Shut down glidein if in Claimed/Idle for too long
Feature #3203: OSG: condor_q -analyze analogue for glideins
Feature #3204: OSG: file transfer plugins in pilot
Feature #3423: Frontend should be able to temporarily blacklist factory entries
Feature #4684: Advertise EC2 info in the startd classads
Feature #4695: Document "check" debug mode
Feature #4846: Include GWMS_XSLT_PLUGIN_DIR setting in generated {factory,frontend}_startup
Feature #4904: Specify a default for GWMS_XSLT_PLUGIN_DIR
Feature #5011: Frontend monitoring should also show what factory is doing with its requests
Feature #5012: Factory monitoring to display sites with issues
Feature #5013: Factory monitoring: factory status from logs should aggregate info per frontend
Feature #5417: analyze_entries should be able to group for FE reports
Feature #5606: Propagating attributes from glideins to factory entries
Feature #5871: If possible reduce the number of secondary collectors in the wmscollector config
Feature #6332: Include files for Frontend config
Feature #8331: Multicore Cloud VMs should provide privilege separation between any two running jobs
Feature #11852: Collect service performance stats every iteration for Factory
Feature #13531: Force the update of all dict files during an upgrade of when format changes
Feature #13875: Ability for facility service providers to add info to USER METADATA
Feature #17256: Create a script to write the results of all he tests in a single table
Feature #17363: Mock condorMonitor functions in unit tests
Feature #18371: Add core counting in factoryEntryStatusNow
Feature #18857: Beautify/imptove javascript code
Feature #18860: Support direct local submission of glideins, e.g. for sites with double authentication or VMs
Feature #19540: Review and improve the behavior of fork.py
Feature #19950: Speed up stop/reconf/start
Feature #20901: Run a shell linter on bash scripts and add unit tests
Feature #21324: Handle new GCE credential format
Feature #21340: Link the 2 existing glideinFrontendElement.py files
Feature #21823: Drop CREAM CE support
Feature #22843: Develop a Testing framework for distributed resources and GlideinWMS
Feature #22940: Transfer current website into Jekyll
Feature #22941: Edit code in Jekyll to align with previously discussed website organization
Feature #23117: Additional logging channel
Feature #23330: Simplifying the support of multiple VOs on one Frontend host
Feature #23584: Add support for ALCF clusters
Feature #24292: TODOs from #22470 related to "with statement"
Feature #25453: Mechanism for file selection in Glidein files
Feature #25454: Add a way to store in GitHub and share the Glidein's VO files
Feature #25505: Support alternative glideins
Feature #25506: Glidein not relying on HTCondor file transfer
Feature #25535: Create a Glidein bundle that includes all the files downloaded from Factory and Frontend to be able to start without network connectivity
Idea #2181: Get Pilot logs back to the factory from an EC2 VM
Idea #3384: Add plugin for APFMon
Idea #23144: Start introducing Python type annotation
Idea #24351: Revise the project directory structure
Milestone #4989: "Why is my job not running"?
Milestone #11850: Collect service performance stats every iteration for Factory & Frontend
Milestone #13397: Investigate the use of condor_annex to provision bulk resources with GlideinWMS
Milestone #23583: Make Frontend code more modular

v3_9_1

GlideinWMS 3.9.1

20%

10 issues   (2 closed — 8 open)

Related issues
Bug #24792: Fix pylint errors as of 8/14
Bug #24940: Check that all path handling is compliant w/ pep-0519 (Path-like, str, bytes handling)
Bug #24960: Change "in list(mydict.keys())" to "in mydict" when possible
Bug #24995: Some divisions are actually float, not meant to be integer
Bug #25008: An error is returned by rrdtool when updating Log_Counts.rrd
Bug #25291: Duplicate glideclient classads - problem with hash() in Python 3
Bug #25416: problems found testing 3.9.1 rc4
Feature #24976: The string conversion is a bit abused, when for "machine consumption" should be repr(), not str()
Feature #25186: Merge v3.7.1, solve conflicts, port the code if needed,

v3_x

Placeholder version for issues

13%

111 issues   (12 closed — 99 open)

Related issues
Bug #2358: Condor for VO Frontend is requesting a cert/proxy, but not using it
Bug #2440: Improve frontend performance
Bug #2441: Accounting issues for new glidein types
Bug #2635: Review glidein cleanup code
Bug #2722: monitoring graph labels
Bug #2944: Re-organize glidein pilot documentation
Bug #3002: Finish documenting the feature from #2455
Bug #3075: glideins not propery configuring the shell
Bug #3103: Overly complicated secondary schedd config
Bug #3118: Change code to implement faster dict lookup
Bug #3125: Rename query_expr and start_expr to *_condor_expr
Bug #3167: FactoryMatchAttrs never used
Bug #3486: GLIDECLIENT_Group_Start when added to attr for rpm install does not populate other attr correctly
Bug #3510: Glideins don't recognize misconfigured Condor
Bug #3556: For partitionable slots, VOFE will stop too early
Bug #3628: Proxy creation plugin needs to be documented
Bug #3641: Reduce the size of release tarball
Bug #3680: Refine the OSG glideinwms documentation
Bug #3702: factoryStatus.html doesn't recognize "frontend=" parameter
Bug #3761: analyze_entries reporting on stale data
Bug #3792: Modify wmsXMLView.py to use findGroupWork rather than findWork from glideFactoryInterface
Bug #3793: Better argument parsing in glidein scritps
Bug #3799: glidecondor_upgrade attempts to perform an upgrade even with invalid arguments
Bug #3810: condor_config from the condor rpms may conflict with the condor_config shipped by glideinwms
Bug #4273: Document factory tool create_condor_tarball
Bug #4339: Fix whitespace for analyze_* tools
Bug #4372: Document factory tool add_entry
Bug #4373: Document factory tool clone_glidein
Bug #4374: Document factory tool info_glidein
Bug #4460: Re-architect the way we do logging in factory and frontend
Bug #4489: Reduce the memory consumption of the factory
Bug #4623: Pilot should reload grid environment between jobs
Bug #4627: manage-glideins does not set system path correctly
Bug #4690: condor_platform_select does not support RHEL/SL 6 OS selection in auto mode
Bug #4814: Get rid of the obsolete vdt pacman related config variables in ini installer
Bug #6370: Submitting glideins using EC2 is hitting 8K limit for URI
Bug #7107: better control over what credentials a frontend passes to the factory
Bug #18520: Cleanup html code
Feature #1925: Factory: GUI for maintenance of glideinWMS.xml configuration
Feature #2000: Validate factory and frontend configurations on service startup
Feature #2037: frontendStatusNow monitoring page
Feature #2180: Use Condor Job Log instead of Schedd log or classads for job data on the pilot
Feature #2192: SImple configurator for RPM installs
Feature #2194: Frontend need for a management mapfile
Feature #2239: Remove multiple glideins at a time
Feature #2258: Adding support for shared_port_daemon for the collector
Feature #2386: Moving the upgrade process to a separate script in the code base
Feature #2410: Frontend monitoring to provide average over 2 hrs, 24 hrs and 7 days
Feature #2411: Frontend better control over number of extra glideins to request
Feature #2413: Changing the location of setup scripts factory.sh and frontend.sh
Feature #2472: Send more VOFE information (via RSL?) to factory
Feature #2511: Eliminate the need for a proxy for VO Frontend communication with the WMS Collector
Feature #2569: Gather worldwide glideinwms usage information
Feature #2594: Native support for glexec in v3+
Feature #2806: Handle Condor host certificate DNS name validation
Feature #2962: Add held job count to factoryStatusNow page
Feature #2964: Cleanly separate the XML handling code in glidein_startup.sh
Feature #3060: Add EVENT_LOG for forensics
Feature #3102: Add a tool to create links to LSB standard locations for Condor tarball installs
Feature #3129: Enhancements for glideFactoryLib.which
Feature #3216: Enhancements for Frontend Monitoring
Feature #3217: Better stale glidein cleanup or factory
Feature #3224: Upgrade Frontend Monitoring
Feature #3229: Add actual idle totals to FE monitoring
Feature #3294: Some code cleanup
Feature #3364: add a filter to the factoryStatusNow webpage
Feature #3429: Add old running glidein info to the factoryStatusNow monitoring page, Troubleshoot view
Feature #3534: We need to find a way to pass the FE controlled values to the Factory.
Feature #3570: Publish items from job.descript in factoryEntryStatusNow
Feature #3679: Testing glexec with a test startd run
Feature #3706: Enabling FE to easily add a file transfer plugin
Feature #3718: Add tool to replace <entries> from central repository
Feature #3818: Add timeouts to validation scripts
Feature #4070: Reduce glideinEntryGroup memory footprint
Feature #4290: Run glideinWMS pilots on ARM achitecture workers
Feature #4435: Advertize ClassAd if factory removes held glidein
Feature #4556: Fall back to curl if wget not found
Feature #4571: Make DN specification optional
Feature #4645: Put pilot logs in subdirectories
Feature #4680: Quantize glidein shutdown times in the cloud
Feature #5309: Need more prompt fake running glidein detection
Feature #5653: Add support for multiple collector partitions
Feature #6195: Add sanity check tools to the Frontend
Feature #6317: Add periodic test glidein submissions
Feature #6319: Propagate and/or validate factory attributes from running glideins
Feature #6891: Tools/Functionality to help with OSG scalability testing
Feature #7117: Perform end-to-end testing using ipv6 testbed
Feature #7119: Treat the RPM as default
Feature #7516: Remove secondary schedd configuration form the default HTCondor factory configuration
Feature #7826: Need additional debugging information in the glideresource classad
Idea #2145: Tracking waste due to preemption in the frontend monitoring
Idea #2418: Better control over removal of glideins
Idea #2592: Factory attribute parsing may be faulty
Idea #2593: Frontend attribute parsing may be faulty
Idea #2688: Adding interactive properties to the factory <-> frontend protocol
Idea #2801: Move factory monitoring aggregation into its own process
Idea #2807: Tracking the number of served requests
Idea #3160: Reducing the memory footprint of the FE with clustering
Idea #3165: Remove condor_q from the factory
Idea #3389: Add a Collector for glidein monitoring to the factory
Idea #3722: Improve debug classad implementation in advertise_failure.helper
Idea #3816: Setting the NegotiatorPort to make it firewall friendly
Idea #5970: Rewrite the table-based Factory Web pages using jQuery libraries
Idea #6257: Minimize web page code duplication
Idea #7187: Revisit GSI_DAEMON_NAME authentication scheme
Idea #7188: PBS Maxwalltime for long running pilots
Milestone #4992: Aggregate monitoring
Milestone #7185: Improve frontend monitoring for Multi Core Glideins