Project

General

Profile

Roadmap

Summary

Roadmap Summary

No issues for this version

RoadmapSummary

  • High priority topics
    • Migration to single-user Factories
      • Maintain or reduce needed operation effort (improvement of Factory scripts)
    • Add support for tokens
      • Use HTCondor token-auth for Glideins authentication
      • Support submission of Glideins to sites using sci-token
      • Authenticate Factory and Frontend using sci-tokens
    • Improve modularity and code quality (especially of Frontend)
      • Move to Python 3
      • Improve code quality
      • Make the code more modular (especially the Frontend)
      • Expand, simplify and automate testing
      • Have a Python 3 version in OSG upcoming by late end of the year
      • Keep supporting Python 2 until OSG 3.4 is supported (probably Spring 2020)
    • Singularity support
      • Singularity invocation via HTCondor
      • Support condor_ssh_to_job for Singularity jobs
      • Allow VO scripts also inside Singularity
    • Factory supporting multiple frontend-like services
      • Improve Decision Engine support
    • Use more of HTCondor features in GWMS
      • Blackhole prevention
      • Singularity invocation via HTCondor
      • Use of tokens (security without x509 certificates)
    • Automatic Factory configuration generation, via CRIC
    • Upgrade Frontend and Factory monitoring
      • Collaborate with UCSD (Edgar) and merge the results in production

v3_6_2

GlideinWMS 3.6.2

5%

40 issues   (2 closed — 38 open)

Related issues
Bug #16702: Fix scripts install/glidecondor_createSecCol and install/glidecondor_createSecSched
Bug #20861: Error message related to entry in the Factory logs
Bug #21832: PicklingError: Can't pickle <type 'Boost.Python.enum'>: import of module Boost.Python failed
Bug #22245: Incompatibility with schedd 8.8.1 and frontend at 8.6
Bug #22623: Print collector name when reconfig fails because of failed communication
Bug #23062: Frontend and Factory reload failing
Bug #23089: fermifactory02 monitoring is wrong somehow
Bug #23341: Total of frontend info is 0 in Factory monitring
Bug #23467: Find CI utility files also in current directory
Bug #23675: Adjust size of ClusterId and ProcessId fields for the condor_q custom format
Bug #23920: Pilot using an expired proxy when authenticating to the collector
Feature #13416: Propagate attributes from the job classad to the glidein JDL
Feature #19947: Do not restart condor on factory upgrade
Feature #19948: Clean up configuration files from disabled entries
Feature #19949: Remove really old files from reconfig.
Feature #20776: Ask for feedback about the new Singularity scripts and remove the TODOs
Feature #20811: Adopt Singularity mechanisms provided by HTCondor
Feature #21711: Add a portable condor_chirp for jobs running under GlideinWMS
Feature #21741: Improve monitoring stats in glidefactory and glidefactoryclient classads
Feature #21875: Invoke Singularity via HTCondor INSTEAD of GWMS job wrapper
Feature #21886: Custom modules for job wrappers
Feature #22163: Check if there are load changes in Factory and solve TODOs added in #21880
Feature #22284: Allow the Singularity wrapper to ensure unicity of bind mounts
Feature #22846: Increase use of HTCondor Python bindings and add them as requirement
Feature #22848: Improve Glidein monitoring and troubleshooting
Feature #22866: Create dynamic pages serving the Glideins stdout, stderr and included content
Feature #23185: Drop tar ball installation of Factory and Frontend
Feature #23290: Support condor_ssh_to_jpb also for Singularity jobs
Feature #23340: Glidein are released when they should not
Feature #23803: Make lgwms-logcat.sh work with non standard factory instance names
Idea #20799: Automate the generation of factory configuration
Idea #23451: Allow a Frontend to run in parallel w/o affecting the Factory

v3_6_3

GlideinWMS 3.6.3

6%

32 issues   (0 closed — 32 open)

Related issues
Bug #15943: Overwriting of /etc/gwms-factory/glideinWMS.xml
Bug #22868: Setting CONDOR_OS to AUTO is not working because of a keywords mismatch
Bug #23038: Directory /var/lib/gwms-frontend/vofrontend/monitor blocks upgrade and reconfig commands
Bug #23176: unit test failures after changing to python htcondor bindings
Bug #23768: Bug in multiple credential distribution
Feature #2121: VO Frontend missing features for Cloud
Feature #2531: Monitoring for frontend: store the number of Job restarts
Feature #10397: Identify means to pass some job related information to the glideins requested
Feature #11500: Support condor_q -autocluster to increase performance
Feature #13854: Migrate to signing type to sha256
Feature #13965: Set up a project repo with the glideinwms rpms
Feature #15280: Support auto for memory just like we do for cores
Feature #18148: Enable proxy extension for Glideins
Feature #18855: Add unit test for lib/fork.py
Feature #19214: Add a configurable limit to the rate of jobs running and fail the glidein if the rate is passed
Feature #20505: Consider if the blacklisting script from FIFE can be added as GlideinWMS frontend feature
Feature #21292: Add GCE termination command as suggested by Burt
Feature #21885: Support to run test and periodic scripts within Singularity
Feature #22036: Dots seem to cause problems in entry names
Feature #22190: Remove client blacklist from the factory
Feature #23053: Add attributes that are VO and site dependent
Feature #23253: Blackhole detection logs: Publishing expression/classAd attributes to the StartdLogs
Feature #23381: Harmonize CI scripts
Feature #23486: Make Glidein's condor resilient to missing or mis-configured hostname
Feature #23578: Review Glidein variables: lifetime, naming, defaults (especially the ones used in Singularity)
Feature #23906: Create a script to prune and purge the web-area

v3_6_x

Placeholder for v3.6 series (OSG 3.4 and 3.5 production) issues

11%

56 issues   (2 closed — 54 open)

Related issues
Bug #2169: factoryEntryStatusNow.html buggy - mixing 3 xml files
Bug #4423: factoryCompletedStats doesn't account for multicore
Bug #4619: Factory info logs message regarding classad advertising
Bug #4813: Incorrect/Misleading sample ini files in the docs
Bug #5241: Frontend credential selection plugin ProxyUserMapWRecycling seems to be broken
Bug #5414: Multiple proxies not being handled correctly - numbers too low
Bug #6991: Document important glidefactory/client/factoryclient classad attributes
Bug #8331: Multicore Cloud VMs should provide privilege separation between any two running jobs
Bug #11142: Collision when 2 credentials use the same file name
Bug #14869: Hardening available cores auto detection
Bug #23187: Active entries not corresponding with the ones defined in the glideinWMS.xml
Feature #2408: Frontend to detect if its mapped name is not correct
Feature #4846: Include GWMS_XSLT_PLUGIN_DIR setting in generated {factory,frontend}_startup
Feature #4904: Specify a default for GWMS_XSLT_PLUGIN_DIR
Feature #5011: Frontend monitoring should also show what factory is doing with its requests
Feature #5012: Factory monitoring to display sites with issues
Feature #5013: Factory monitoring: factory status from logs should aggregate info per frontend
Feature #5345: Expand attribute expansion capabilities
Feature #5417: analyze_entries should be able to group for FE reports
Feature #5606: Propagating attributes from glideins to factory entries
Feature #5871: If possible reduce the number of secondary collectors in the wmscollector config
Feature #6332: Include files for Frontend config
Feature #6556: Frontend massively overprovisions sites due to no grouping of entries
Feature #7815: Manage OOM score of frontend processes
Feature #11754: Allow controlling of site/entry's limits from the frontend
Feature #11756: Is it time to get rid of TotalTimeUnclaimedIdle?
Feature #11802: Ability to add custom information to the EC2 USER META DATA
Feature #11852: Collect service performance stats every iteration for Factory
Feature #13531: Force the update of all dict files during an upgrade of when format changes
Feature #13875: Ability for facility service providers to add info to USER METADATA
Feature #17256: Create a script to write the results of all he tests in a single table
Feature #17363: Mock condorMonitor functions in unit tests
Feature #18042: Change exit code of pylint and unit test scripts
Feature #18371: Add core counting in factoryEntryStatusNow
Feature #18857: Beautify/imptove javascript code
Feature #18860: Support direct local submission of glideins, e.g. for sites with double authentication or VMs
Feature #19540: Review and improve the behavior of fork.py
Feature #19950: Speed up stop/reconf/start
Feature #20237: Design a logo for GlideinWMS
Feature #21313: Containerize Factory deployment
Feature #21314: Containerize Frontend deployment
Feature #21324: Handle new GCE credential format
Feature #21340: Link the 2 existing glideinFrontendElement.py files
Feature #22843: Develop a Testing framework for distributed resources and GlideinWMS
Feature #22940: Transfer current website into Jekyll
Feature #22941: Edit code in Jekyll to align with previously discussed website organization
Feature #22988: Move files to be accessed and tracked by others
Feature #23330: Simplifying the support of multiple VOs on one Frontend host
Idea #23144: Start introducing Python type annotation
Milestone #4989: "Why is my job not running"?
Milestone #11850: Collect service performance stats every iteration for Factory & Frontend
Milestone #13397: Investigate the use of condor_annex to provision bulk resources with GlideinWMS

v3_7

GlideinWMS v3_7

22%

12 issues   (0 closed — 12 open)

Related issues
Bug #21989: Make attribute publishing independent from other properties
Bug #23052: parameter=True ignored in Factory attributes if const=True or publish=False
Bug #23339: Make wholenode knobs uniform
Bug #23376: Possible bug if there are extra spaces at the beginning or at the end of a value in glidein_config
Feature #20901: Run a shell linter on bash scripts and add unit tests
Feature #21558: Add the ability to advertise an attribute in the entry classad also if constant
Feature #23091: Reliable, flexible and secure logging system for distributed workflows
Feature #23092: Use token-auth for Glideins authentication and add support for sci-token
Feature #23136: Add an attribute to save the logging channel content in the Glidein stderr
Feature #23265: Add security mechanisms to the glidein logging channel

v3_7_1

GlideinWMS 3.7.1

No issues for this version

v3_7_x

Placeholder for v3.7 development series (OSG 3.5 upcoming) issues

8%

22 issues   (0 closed — 22 open)

Related issues
Bug #4050: Factory should handle os.fork() failing
Bug #4536: Problem advertising requests affecting FE logs
Bug #4587: OSG: Automate / fix HTCondor-CE issues
Bug #4589: Init script "restart" action incorrect
Bug #4617: Race in TERM handling in glideinFrontendElement
Bug #4678: Glidein reports negative retire times
Bug #4781: Secondary Condor Collectors share address files
Bug #4899: Cloud environment doesn't source profile.d
Bug #7848: If num_parallel_workers are 1 and a child crashes, factory can get stuck
Bug #23612: Make sure that the classad attribute names are treated as case insensitive
Feature #2897: Shut down glidein if in Claimed/Idle for too long
Feature #3203: OSG: condor_q -analyze analogue for glideins
Feature #3204: OSG: file transfer plugins in pilot
Feature #3423: Frontend should be able to temporarily blacklist factory entries
Feature #4684: Advertise EC2 info in the startd classads
Feature #4695: Document "check" debug mode
Feature #21823: Drop CREAM CE support
Feature #23117: Additional logging channel
Feature #23584: Add support for ALCF clusters
Idea #2181: Get Pilot logs back to the factory from an EC2 VM
Idea #3384: Add plugin for APFMon
Milestone #23583: Make Frontend code more modular

v3_9

GlideinWMS 3.9

No issues for this version

v3_9_x

Placeholder for v3.9 series (Python 3 GWMS) issues

0%

3 issues   (0 closed — 3 open)

Related issues
Feature #20232: Run unit tests in Python 3
Feature #20766: Change configuration handling and configuration files format
Idea #21559: Revise the attribute (attr) and submit_attribute system

v3_x

Placeholder version for issues

8%

113 issues   (6 closed — 107 open)

Related issues
Bug #2358: Condor for VO Frontend is requesting a cert/proxy, but not using it
Bug #2440: Improve frontend performance
Bug #2441: Accounting issues for new glidein types
Bug #2635: Review glidein cleanup code
Bug #2722: monitoring graph labels
Bug #2944: Re-organize glidein pilot documentation
Bug #3002: Finish documenting the feature from #2455
Bug #3075: glideins not propery configuring the shell
Bug #3103: Overly complicated secondary schedd config
Bug #3118: Change code to implement faster dict lookup
Bug #3125: Rename query_expr and start_expr to *_condor_expr
Bug #3167: FactoryMatchAttrs never used
Bug #3486: GLIDECLIENT_Group_Start when added to attr for rpm install does not populate other attr correctly
Bug #3510: Glideins don't recognize misconfigured Condor
Bug #3556: For partitionable slots, VOFE will stop too early
Bug #3628: Proxy creation plugin needs to be documented
Bug #3641: Reduce the size of release tarball
Bug #3680: Refine the OSG glideinwms documentation
Bug #3702: factoryStatus.html doesn't recognize "frontend=" parameter
Bug #3761: analyze_entries reporting on stale data
Bug #3792: Modify wmsXMLView.py to use findGroupWork rather than findWork from glideFactoryInterface
Bug #3793: Better argument parsing in glidein scritps
Bug #3799: glidecondor_upgrade attempts to perform an upgrade even with invalid arguments
Bug #3810: condor_config from the condor rpms may conflict with the condor_config shipped by glideinwms
Bug #4273: Document factory tool create_condor_tarball
Bug #4339: Fix whitespace for analyze_* tools
Bug #4372: Document factory tool add_entry
Bug #4373: Document factory tool clone_glidein
Bug #4374: Document factory tool info_glidein
Bug #4460: Re-architect the way we do logging in factory and frontend
Bug #4489: Reduce the memory consumption of the factory
Bug #4623: Pilot should reload grid environment between jobs
Bug #4627: manage-glideins does not set system path correctly
Bug #4690: condor_platform_select does not support RHEL/SL 6 OS selection in auto mode
Bug #4814: Get rid of the obsolete vdt pacman related config variables in ini installer
Bug #6370: Submitting glideins using EC2 is hitting 8K limit for URI
Bug #7107: better control over what credentials a frontend passes to the factory
Bug #7243: Frontend support for HA Collectors
Bug #18520: Cleanup html code
Feature #1925: Factory: GUI for maintenance of glideinWMS.xml configuration
Feature #2000: Validate factory and frontend configurations on service startup
Feature #2037: frontendStatusNow monitoring page
Feature #2180: Use Condor Job Log instead of Schedd log or classads for job data on the pilot
Feature #2192: SImple configurator for RPM installs
Feature #2194: Frontend need for a management mapfile
Feature #2239: Remove multiple glideins at a time
Feature #2258: Adding support for shared_port_daemon for the collector
Feature #2386: Moving the upgrade process to a separate script in the code base
Feature #2410: Frontend monitoring to provide average over 2 hrs, 24 hrs and 7 days
Feature #2411: Frontend better control over number of extra glideins to request
Feature #2413: Changing the location of setup scripts factory.sh and frontend.sh
Feature #2472: Send more VOFE information (via RSL?) to factory
Feature #2511: Eliminate the need for a proxy for VO Frontend communication with the WMS Collector
Feature #2569: Gather worldwide glideinwms usage information
Feature #2594: Native support for glexec in v3+
Feature #2806: Handle Condor host certificate DNS name validation
Feature #2962: Add held job count to factoryStatusNow page
Feature #2964: Cleanly separate the XML handling code in glidein_startup.sh
Feature #3060: Add EVENT_LOG for forensics
Feature #3102: Add a tool to create links to LSB standard locations for Condor tarball installs
Feature #3129: Enhancements for glideFactoryLib.which
Feature #3216: Enhancements for Frontend Monitoring
Feature #3217: Better stale glidein cleanup or factory
Feature #3224: Upgrade Frontend Monitoring
Feature #3229: Add actual idle totals to FE monitoring
Feature #3294: Some code cleanup
Feature #3364: add a filter to the factoryStatusNow webpage
Feature #3429: Add old running glidein info to the factoryStatusNow monitoring page, Troubleshoot view
Feature #3534: We need to find a way to pass the FE controlled values to the Factory.
Feature #3570: Publish items from job.descript in factoryEntryStatusNow
Feature #3679: Testing glexec with a test startd run
Feature #3706: Enabling FE to easily add a file transfer plugin
Feature #3718: Add tool to replace <entries> from central repository
Feature #3818: Add timeouts to validation scripts
Feature #4070: Reduce glideinEntryGroup memory footprint
Feature #4290: Run glideinWMS pilots on ARM achitecture workers
Feature #4435: Advertize ClassAd if factory removes held glidein
Feature #4556: Fall back to curl if wget not found
Feature #4571: Make DN specification optional
Feature #4645: Put pilot logs in subdirectories
Feature #4680: Quantize glidein shutdown times in the cloud
Feature #5309: Need more prompt fake running glidein detection
Feature #5652: Add support for flocked collectors
Feature #5653: Add support for multiple collector partitions
Feature #6195: Add sanity check tools to the Frontend
Feature #6317: Add periodic test glidein submissions
Feature #6319: Propagate and/or validate factory attributes from running glideins
Feature #6891: Tools/Functionality to help with OSG scalability testing
Feature #7117: Perform end-to-end testing using ipv6 testbed
Feature #7119: Treat the RPM as default
Feature #7516: Remove secondary schedd configuration form the default HTCondor factory configuration
Feature #7826: Need additional debugging information in the glideresource classad
Idea #2145: Tracking waste due to preemption in the frontend monitoring
Idea #2418: Better control over removal of glideins
Idea #2592: Factory attribute parsing may be faulty
Idea #2593: Frontend attribute parsing may be faulty
Idea #2688: Adding interactive properties to the factory <-> frontend protocol
Idea #2801: Move factory monitoring aggregation into its own process
Idea #2807: Tracking the number of served requests
Idea #3160: Reducing the memory footprint of the FE with clustering
Idea #3165: Remove condor_q from the factory
Idea #3389: Add a Collector for glidein monitoring to the factory
Idea #3722: Improve debug classad implementation in advertise_failure.helper
Idea #3816: Setting the NegotiatorPort to make it firewall friendly
Idea #5970: Rewrite the table-based Factory Web pages using jQuery libraries
Idea #6257: Minimize web page code duplication
Idea #7187: Revisit GSI_DAEMON_NAME authentication scheme
Idea #7188: PBS Maxwalltime for long running pilots
Milestone #4992: Aggregate monitoring
Milestone #7185: Improve frontend monitoring for Multi Core Glideins