Feature #22843
Develop a Testing framework for distributed resources and GlideinWMS
0%
Description
Build a testing application for distributed resources (entries) and GlideinWMS.
This application will interact with GlideinWMS components and send Glideins to explore and validate the resources.
It will further allow to customize the Glideins and replace standard components of the GlideinWMS framework with new ones than the user decides to test.
- Provide input in the design of the testing application
- Getting familiar with distributed computing and GlideinWMS
- Development and integration related to distributed computing software for Grids, Clouds and Supercomputers (this will include Python and Web development):
- Adapting components to talk the GlideinWMS protocol
- Adding a UI to drive the testing
- Testing on clusters, High-Performance Computers and clouds
- Integration and release
- No flexible way to send pilots to specific resources
- No way to substitute standard scripts
- Manual client checking which resources are available, what's the status and letting the user decide how many pilots to send and where
- The code written should be in Python 3 (specifically 3.6)
- Consider migrating some of the code in GlideinWMS (e.g. protocol sending ClassAds) using python 3.6 and htcondor python bindings
Here a project doing something similar with shell scripts:
https://cms-htcondor-monitor.web.cern.ch/cms-htcondor-monitor/_book/emulated_pilot/
https://gitlab.cern.ch/CMSSI/cms_emulated_glidein
Related issues
History
#1 Updated by Marco Mambelli almost 2 years ago
- Related to Milestone #22673: Summer interns 2019 added
#2 Updated by Javier Rodriguez almost 2 years ago
- % Done changed from 0 to 70
#3 Updated by Javier Rodriguez almost 2 years ago
- % Done changed from 70 to 100
- Status changed from New to Resolved
#4 Updated by Javier Rodriguez almost 2 years ago
- % Done changed from 100 to 80
- Status changed from Resolved to Assigned
#5 Updated by Marco Mambelli over 1 year ago
Here a concrete task:
Start with manual_glidein_submit.py
Add the possibility to modify the parameters used in the submission (params dictionary) by reading them from a file in INI format. The file name will be specified in a command option, e.g. --parameters-file FILE_NAME , where FILE_NAME is the path to the INI file
Suggestion:
https://docs.python.org/2/library/configparser.html
Step 2 (more difficult)
Use the YAML format instead of INI and run the manual_glidein_submit with python3.
This may require some changes in the GlideinWMS code imported.
https://learn.getgrav.org/16/advanced/yaml
https://pyyaml.org/wiki/PyYAMLDocumentation
#6 Updated by Marco Mambelli over 1 year ago
- % Done changed from 80 to 0
- Target version set to v3_5_x
- Assignee changed from Javier Rodriguez to Marco Mambelli
- Status changed from Assigned to New
Javier worked on a change to manual_glidein_submit.py to allow extra attr options loaded from an ini file.
The ticket is being opened for future re-assignment
#7 Updated by Marco Mambelli over 1 year ago
- Target version changed from v3_5_x to v3_6_x
#8 Updated by Marco Mambelli 2 months ago
- Target version changed from v3_6_x to v3_7_x