Project

General

Profile

Roadmap

v3_5

about 6 months late (11/30/2018)

GlideinWMS 3.5

9%

23 issues   (2 closed — 21 open)

Related issues
Bug #22282: Remove use of deprecated sets module
Bug #22358: systemctl status reports the wrong status
Bug #22370: Periodic scripts seem to use the prefix inconsistently, only when invoked by startd cron
Bug #22483: pip errors in nightly CI
Bug #22518: Entries in downtime setting seems not to work correctly
Bug #22541: typos in help for release.py
Bug #22542: incorrect exception handling glideinFrontendElement.py::populate_pubkey()
Feature #20215: Remove dependency on condor_root_switchboard
Feature #20545: Problems with the default 'frontend' user in the Factory
Feature #20776: Ask for feedback about the new Singularity scripts and remove the TODOs
Feature #20811: Adopt Singularity mechanisms provided by HTCondor
Feature #21174: Double-check add_entry script no longer needed
Feature #21247: Remove support for Globus GRAM (GT2/GT5)
Feature #22163: Check if there are load changes in Factory and solve TODOs added in #21880
Feature #22190: Remove client blacklist from the factory
Feature #22284: Allow the Singularity wrapper to ensure unicity of bind mounts
Feature #22599: Improve osg-release.sh to show more errors

Summary

Roadmap Summary

No issues for this version

RoadmapSummary

  • High priority topics
    • Move to Python 3
      • Branch with Python 3 migration
      • Have a Python 3 version in OSG upcoming by late Summer 2019
      • Keep supporting Python 2 until OSG 3.4 is supported (probably Spring 2020)
    • Factory supporting multiple frontend-like services
      • Decision Engine support started in 3.4.4
    • Collaboration with HTCondor
      • Blackhole prevention (3.5)
      • Singularity invocation via HTCondor (3.5)
      • Use of tokens (security without x509 certificates)
    • Automatic Factory configuration generation, via CRIC (3.5)
    • Upgrade Frontend and Factory monitoring
      • Collaborate with UCSD (Edgar) and merge the results in production

v3_4_4

GlideinWMS 3.4.4

94%

16 issues   (15 closed — 1 open)

Related issues
Bug #21682: Glidein not killing condor processes
Bug #21880: Factory GlideinMonitor* classads appear to be erased periodically
Bug #21898: Error preventing the Frontend fo match jobs
Bug #21980: Make sure metasite limits are respected
Bug #21982: Pilot proxies could be generated with invalid VOMS ACs
Bug #22022: Convert per/frontend limits to integers before division
Bug #22034: 'tochild' AttributeError in Factory due to leftover popen method
Feature #3532: Propagate to Factory and glidein submission attributes controlled by FE
Feature #15176: Track jobs that spawn multiple nodes, e.g. HPC submission
Feature #17560: Include unprivileged singularity in pilot software
Feature #21536: unit tests ready for review
Feature #21639: Include OSG distributed unprivileged Singularity to the search path and do a full test of Singularity
Feature #21949: SCHEDDGLIDEINS2_SPOOL_DIR_STRING in comment is confusing in the condor configuration in the Factory
Feature #22196: Ignore entries in downtime when considering glideins matches
Feature #22277: The Frontend service in SL7 always require the HTCondor service. This is not true if there is a standalone frontend
Feature #22285: Add a bind mount for /etc/hosts in Singularity

v3_4_x

Placeholder for v3.4 series issues

No issues for this version

v3_5_1

GlideinWMS v3_5_1

2%

54 issues   (0 closed — 54 open)

Related issues
Bug #9639: Glidein startup aborting fast for apparently no reason (in reality conor_master is timing out a name resolution)
Bug #15943: Overwriting of /etc/gwms-factory/glideinWMS.xml
Bug #16702: Fix scripts install/glidecondor_createSecCol and install/glidecondor_createSecSched
Bug #20861: Error message related to entry in the Factory logs
Bug #21354: Unittests failing at times on SL7
Bug #21989: Make attribute publishing independent from other properties
Bug #22245: Incompatibility with schedd 8.8.1 and frontend at 8.6
Bug #22623: Print collector name when reconfig fails because of failed communication
Feature #2121: VO Frontend missing features for Cloud
Feature #2531: Monitoring for frontend: store the number of Job restarts
Feature #10397: Identify means to pass some job related information to the glideins requested
Feature #11500: Support condor_q -autocluster to increase performance
Feature #12838: Provide flexible mechanism for custom scripts to log information that can be brought back along with the other condor logs
Feature #13416: Propagate attributes from the job classad to the glidein JDL
Feature #13854: Migrate to signing type to sha256
Feature #13965: Set up a project repo with the glideinwms rpms
Feature #15280: Support auto for memory just like we do for cores
Feature #18371: Add core counting in factoryEntryStatusNow
Feature #18855: Add unit test for lib/fork.py
Feature #19214: Add a configurable limit to the rate of jobs running and fail the glidein if the rate is passed
Feature #19947: Do not restart condor on factory upgrade
Feature #19948: Clean up configuration files from disabled entries
Feature #19949: Remove really old files from reconfig.
Feature #19950: Speed up stop/reconf/start
Feature #20232: Run unit tests in Python 3
Feature #20505: Consider if the blacklisting script from FIFE can be added as GlideinWMS frontend feature
Feature #20900: Code improvement in CPU evaluation/detection
Feature #20901: Run a shell linter on bash scripts and add unit tests
Feature #21292: Add GCE termination command as suggested by Burt
Feature #21340: Ljnk the 2 existing glideinFrontendElement.py files
Feature #21558: Add the ability to advertise an attribute in the entry classad also if constant
Feature #21711: Add a portable condor_chirp for jobs running under GlideinWMS
Feature #21741: Improve monitoring stats in glidefactory and glidefactorystatus classads
Feature #21875: Invoke Singularity via HTCondor instead of GWMS job wrapper
Feature #21885: Support to run test and periodic scripts within Singularity
Feature #22036: Dots seem to cause problems in entry names
Feature #22579: Support https URLs
Idea #20033: Should tarball installations be supported?
Idea #20799: Automate the generation of factory configuration
Idea #22470: Use of Python constructs when possible to help context managers to properly manage resources.

v3_5_x

Placeholder for v3.5 series (Single user Factory) issues

5%

66 issues   (0 closed — 66 open)

Related issues
Bug #2169: factoryEntryStatusNow.html buggy - mixing 3 xml files
Bug #3687: Fix documentation in v3
Bug #4050: Factory should handle os.fork() failing
Bug #4423: factoryCompletedStats doesn't account for multicore
Bug #4536: Problem advertising requests affecting FE logs
Bug #4587: OSG: Automate / fix HTCondor-CE issues
Bug #4589: Init script "restart" action incorrect
Bug #4617: Race in TERM handling in glideinFrontendElement
Bug #4619: Factory info logs message regarding classad advertising
Bug #4678: Glidein reports negative retire times
Bug #4781: Secondary Condor Collectors share address files
Bug #4813: Incorrect/Misleading sample ini files in the docs
Bug #4899: Cloud environment doesn't source profile.d
Bug #5241: Frontend credential selection plugin ProxyUserMapWRecycling seems to be broken
Bug #5414: Multiple proxies not being handled correctly - numbers too low
Bug #6991: Document important glidefactory/client/factoryclient classad attributes
Bug #7848: If num_parallel_workers are 1 and a child crashes, factory can get stuck
Bug #8331: Multicore Cloud VMs should provide privilege separation between any two running jobs
Bug #11142: Collision when 2 credentials use the same file name
Bug #14869: Hardening available cores auto detection
Bug #20202: Kill not handled properly by glidein_startup.sh
Feature #2408: Frontend to detect if its mapped name is not correct
Feature #2897: Shut down glidein if in Claimed/Idle for too long
Feature #3203: OSG: condor_q -analyze analogue for glideins
Feature #3204: OSG: file transfer plugins in pilot
Feature #3423: Frontend should be able to temporarily blacklist factory entries
Feature #4684: Advertise EC2 info in the startd classads
Feature #4695: Document "check" debug mode
Feature #4846: Include GWMS_XSLT_PLUGIN_DIR setting in generated {factory,frontend}_startup
Feature #4904: Specify a default for GWMS_XSLT_PLUGIN_DIR
Feature #5011: Frontend monitoring should also show what factory is doing with its requests
Feature #5012: Factory monitoring to display sites with issues
Feature #5013: Factory monitoring: factory status from logs should aggregate info per frontend
Feature #5345: Expand attribute expansion capabilities
Feature #5417: analyze_entries should be able to group for FE reports
Feature #5606: Propagating attributes from glideins to factory entries
Feature #5871: If possible reduce the number of secondary collectors in the wmscollector config
Feature #6332: Include files for Frontend config
Feature #6556: Frontend massively overprovisions sites due to no grouping of entries
Feature #7815: Manage OOM score of frontend processes
Feature #11754: Allow controlling of site/entry's limits from the frontend
Feature #11756: Is it time to get rid of TotalTimeUnclaimedIdle?
Feature #11802: Ability to add custom information to the EC2 USER META DATA
Feature #11852: Collect service performance stats every iteration for Factory
Feature #13531: Force the update of all dict files during an upgrade of when format changes
Feature #13875: Ability for facility service providers to add info to USER METADATA
Feature #14586: Write a page to document the meaning of glidein, jobs, slots and similar concepts
Feature #17256: Create a script to write the results of all he tests in a single table
Feature #17363: Mock condorMonitor functions in unit tests
Feature #18042: Change exit code of pylint and unit test scripts
Feature #18857: Beautify/imptove javascript code
Feature #18860: Support direct local submission of glideins, e.g. for sites with double authentication or VMs
Feature #19540: Review and improve the behavior of fork.py
Feature #20237: Design a logo for GlideinWMS
Feature #21313: Containerize Factory deployment
Feature #21314: Containerize Frontend deployment
Feature #21324: Handle new GCE credential format
Feature #21886: Custom modules for job wrappers
Idea #2181: Get Pilot logs back to the factory from an EC2 VM
Idea #3384: Add plugin for APFMon
Milestone #4989: "Why is my job not running"?
Milestone #11850: Collect service performance stats every iteration for Factory & Frontend
Milestone #13397: Investigate the use of condor_annex to provision bulk resources with GlideinWMS

v3_6

GlideinWMS v3_6

No issues for this version

v3_7_x

Placeholder for v3.7 series (Python 3 GWMS) issues

0%

2 issues   (0 closed — 2 open)

Related issues
Feature #20766: Change configuration handling and configuration files format
Idea #21559: Revise the attribute (attr) and submit_attribute system

v3_x

Placeholder version for issues

8%

113 issues   (6 closed — 107 open)

Related issues
Bug #2358: Condor for VO Frontend is requesting a cert/proxy, but not using it
Bug #2440: Improve frontend performance
Bug #2441: Accounting issues for new glidein types
Bug #2635: Review glidein cleanup code
Bug #2722: monitoring graph labels
Bug #2944: Re-organize glidein pilot documentation
Bug #3002: Finish documenting the feature from #2455
Bug #3075: glideins not propery configuring the shell
Bug #3103: Overly complicated secondary schedd config
Bug #3118: Change code to implement faster dict lookup
Bug #3125: Rename query_expr and start_expr to *_condor_expr
Bug #3167: FactoryMatchAttrs never used
Bug #3486: GLIDECLIENT_Group_Start when added to attr for rpm install does not populate other attr correctly
Bug #3510: Glideins don't recognize misconfigured Condor
Bug #3556: For partitionable slots, VOFE will stop too early
Bug #3628: Proxy creation plugin needs to be documented
Bug #3641: Reduce the size of release tarball
Bug #3680: Refine the OSG glideinwms documentation
Bug #3702: factoryStatus.html doesn't recognize "frontend=" parameter
Bug #3761: analyze_entries reporting on stale data
Bug #3792: Modify wmsXMLView.py to use findGroupWork rather than findWork from glideFactoryInterface
Bug #3793: Better argument parsing in glidein scritps
Bug #3799: glidecondor_upgrade attempts to perform an upgrade even with invalid arguments
Bug #3810: condor_config from the condor rpms may conflict with the condor_config shipped by glideinwms
Bug #4273: Document factory tool create_condor_tarball
Bug #4339: Fix whitespace for analyze_* tools
Bug #4372: Document factory tool add_entry
Bug #4373: Document factory tool clone_glidein
Bug #4374: Document factory tool info_glidein
Bug #4460: Re-architect the way we do logging in factory and frontend
Bug #4489: Reduce the memory consumption of the factory
Bug #4623: Pilot should reload grid environment between jobs
Bug #4627: manage-glideins does not set system path correctly
Bug #4690: condor_platform_select does not support RHEL/SL 6 OS selection in auto mode
Bug #4814: Get rid of the obsolete vdt pacman related config variables in ini installer
Bug #6370: Submitting glideins using EC2 is hitting 8K limit for URI
Bug #7107: better control over what credentials a frontend passes to the factory
Bug #7243: Frontend support for HA Collectors
Bug #18520: Cleanup html code
Feature #1925: Factory: GUI for maintenance of glideinWMS.xml configuration
Feature #2000: Validate factory and frontend configurations on service startup
Feature #2037: frontendStatusNow monitoring page
Feature #2180: Use Condor Job Log instead of Schedd log or classads for job data on the pilot
Feature #2192: SImple configurator for RPM installs
Feature #2194: Frontend need for a management mapfile
Feature #2239: Remove multiple glideins at a time
Feature #2258: Adding support for shared_port_daemon for the collector
Feature #2386: Moving the upgrade process to a separate script in the code base
Feature #2410: Frontend monitoring to provide average over 2 hrs, 24 hrs and 7 days
Feature #2411: Frontend better control over number of extra glideins to request
Feature #2413: Changing the location of setup scripts factory.sh and frontend.sh
Feature #2472: Send more VOFE information (via RSL?) to factory
Feature #2511: Eliminate the need for a proxy for VO Frontend communication with the WMS Collector
Feature #2569: Gather worldwide glideinwms usage information
Feature #2594: Native support for glexec in v3+
Feature #2806: Handle Condor host certificate DNS name validation
Feature #2962: Add held job count to factoryStatusNow page
Feature #2964: Cleanly separate the XML handling code in glidein_startup.sh
Feature #3060: Add EVENT_LOG for forensics
Feature #3102: Add a tool to create links to LSB standard locations for Condor tarball installs
Feature #3129: Enhancements for glideFactoryLib.which
Feature #3216: Enhancements for Frontend Monitoring
Feature #3217: Better stale glidein cleanup or factory
Feature #3224: Upgrade Frontend Monitoring
Feature #3229: Add actual idle totals to FE monitoring
Feature #3294: Some code cleanup
Feature #3364: add a filter to the factoryStatusNow webpage
Feature #3429: Add old running glidein info to the factoryStatusNow monitoring page, Troubleshoot view
Feature #3534: We need to find a way to pass the FE controlled values to the Factory.
Feature #3570: Publish items from job.descript in factoryEntryStatusNow
Feature #3679: Testing glexec with a test startd run
Feature #3706: Enabling FE to easily add a file transfer plugin
Feature #3718: Add tool to replace <entries> from central repository
Feature #3818: Add timeouts to validation scripts
Feature #4070: Reduce glideinEntryGroup memory footprint
Feature #4290: Run glideinWMS pilots on ARM achitecture workers
Feature #4435: Advertize ClassAd if factory removes held glidein
Feature #4556: Fall back to curl if wget not found
Feature #4571: Make DN specification optional
Feature #4645: Put pilot logs in subdirectories
Feature #4680: Quantize glidein shutdown times in the cloud
Feature #5309: Need more prompt fake running glidein detection
Feature #5652: Add support for flocked collectors
Feature #5653: Add support for multiple collector partitions
Feature #6195: Add sanity check tools to the Frontend
Feature #6317: Add periodic test glidein submissions
Feature #6319: Propagate and/or validate factory attributes from running glideins
Feature #6891: Tools/Functionality to help with OSG scalability testing
Feature #7117: Perform end-to-end testing using ipv6 testbed
Feature #7119: Treat the RPM as default
Feature #7516: Remove secondary schedd configuration form the default HTCondor factory configuration
Feature #7826: Need additional debugging information in the glideresource classad
Idea #2145: Tracking waste due to preemption in the frontend monitoring
Idea #2418: Better control over removal of glideins
Idea #2592: Factory attribute parsing may be faulty
Idea #2593: Frontend attribute parsing may be faulty
Idea #2688: Adding interactive properties to the factory <-> frontend protocol
Idea #2801: Move factory monitoring aggregation into its own process
Idea #2807: Tracking the number of served requests
Idea #3160: Reducing the memory footprint of the FE with clustering
Idea #3165: Remove condor_q from the factory
Idea #3389: Add a Collector for glidein monitoring to the factory
Idea #3722: Improve debug classad implementation in advertise_failure.helper
Idea #3816: Setting the NegotiatorPort to make it firewall friendly
Idea #5970: Rewrite the table-based Factory Web pages using jQuery libraries
Idea #6257: Minimize web page code duplication
Idea #7187: Revisit GSI_DAEMON_NAME authentication scheme
Idea #7188: PBS Maxwalltime for long running pilots
Milestone #4992: Aggregate monitoring
Milestone #7185: Improve frontend monitoring for Multi Core Glideins