Wiki » History » Version 84
« Previous -
Version 84/137
(diff) -
Next » -
Current version
Thomas Hein, 07/21/2017 09:56 AM
For Stakeholder information, see StakeholderInfo¶
For New Member Orientation, see NewMemberOrientation¶
Meetings Meetings¶
- Table of contents
- For Stakeholder information, see StakeholderInfo
- For New Member Orientation, see NewMemberOrientation
- Meetings Meetings
- Gaining access to the glideinWMS code
- Continuous Integration (CI) using Jenkins
- Making Changes to the code
- Making a Glideinwms Release
- Creating the Glideinwms RPMs
- Installing the Glideinwms RPMs
- Testing on Fermicloud
- Glideinwms On Cloud
- Glideinwms Accounting Improvements
- Removing the big binaries from the tree
- Cleaning Remote Branches
Gaining access to the glideinWMS code¶
To gain write access to the code, you need -
- A valid Fermilab Kerberos principal * A valid services account * Developer level access to the glideinwms project in redmine.fnal.gov
Once you have the access you can checkout the code using
git clone ssh://p-glideinwms@cdcvs.fnal.gov/cvs/projects/glideinwms
NOTE If your authentication is failing (Permission denied ...) even if you have a valid Kerberos ticket, make sure that SSL is configured to use Kerberos.
E.g. in ~/.ssh/config you have:
GSSAPIAuthentication yes GSSAPIDelegateCredentials yes
Continuous Integration (CI) using Jenkins¶
NOTE: GlideinWMS related Jenkins projects are parameterized. Do not change the project configuration in the Jenkins unless you know what you are doing. Doing so may break automated tasks.
Getting Access to the Jenkins CI¶
- Open a service desk ticket directed to the Build services group.
- Use your services account to get a FNAL CI-Logon certificate.
- Upload your certificate into your browser to access the CI service.
Setup Your Account to Trigger Builds/Tests Using CURL¶
- This step is optional but lets you trigger Jenkins jobs remotely
- Login into: https://buildmaster.fnal.gov
- Generate API Token for remote builds/tests using CURL
Pylint & PEP8 Validation Tests¶
Jenkins Project: https://buildmaster.fnal.gov/job/GlideinWMS-Pylint
Daily Automated Tests¶
- Daily tests are run from GlideinWMS GIT project account as a crontab. p-glideinwms@cdcvs.fnal.gov
Running Build/Test Jobs Manually¶
Using Jenkins Webpage
- Login into: https://buildmaster.fnal.gov
- Go the project you want to run
- Click "Build With Parameters" and input the parameters you want to build with. For example, GIT branch(es), mail address to notify on completion, etc
Using CURL
# Sample Command # Project Auth Token: This is available from the project configuration page if you have access to the Jenkins curl -X POST "https://<username>:<API Token>@buildmaster.fnal.gov/job/GlideinWMS-Pylint/buildWithParameters?token=<Project Auth Token>&GWMS_BRANCH=<branches>&GWMS_MAILTO=<email address to notify>"
Making Changes to the code¶
Supported versions¶
At the moment 5/19/17 there are no known running v2.x instances.
In the code there are still sections referring to v2 protocol.
v2 is deprecated, so it is OK not to add new features to code referring only to v2.
Still function calls should not be broken, i.e. the interface of v2 code should be updated.
Code Branching Guidelines¶
To Branch or Not To Branch¶
You should always create a development branch for your work. Creating a branch may not be needed for very minor change to docs. Here are some of the guidelines to make this decision.
Create a Branch when ...
- Adding new feature(s) to the code * There is a possibility of incompatibility * Multiple developers are working on the feature(s) in collaborative manner
Branching and Merging Tips¶
- Create the branch with the branch name that can be easily associated with you or the ticket. Preferred branch naming convention is v3/xxxx where v3 => this ticket is associated with glideinwms v3.x and xxxx => ticket number * Make your changes, test them * Only when you are convinced that your changes are tested and working, assign the ticket to another team member for feedback * If feedback is ok only then merge them back to the respective version branch and/or head. * Remember to update the tags.txt file describing the feature/bug fix * Test the changes after doing the merge to make sure you do not break anything or create incompatibility.
Handling pull requests from collaborators¶
- Changes should be given to you in form of a github pull request with the remote branch name
# STEP 1: Checkout the changes to redmine project repo git checkout <redminerepo_branch_to_fork_from> git checkout -b <redminerepo_patch_branch> git pull https://github.com/<github_patch_provider_username>/glideinWMS.git <github_patched_branch> # STEP 2: TEST THE CHANGES # STEP 3: Merge the changes and push to redmine git checkout <redminerepo_branch_to_merge_into> git merge --no-ff <redminerepo_patch_branch> git push -u origin <redminerepo_patch_branch>
Code Formatting Guidelines¶
Formatting the Code¶
- Use standard libraries as much as possible
* Try avoiding the deprecated API's
* Never use tabs for indentation. Use spaces for indentation.
* Standard python code has 4 spaces for indentation, so use it.
* When in doubt always check with
http://www.python.org/dev/peps/pep-0008/
http://www.python.org/dev/peps/pep-0257 * Use the alias of a type when available (ie. "float" over "types.FloatType". * To check if a variable is of a certain type, use isinstance(variable, type). * Don't raise a message directory, pass a message to another function (ie. raise ValueError("My message") * "has_key()" has been depreciated and is not available in Python v3. Use "in" instead: key_var in dictionary_var. * For more visit Code Formatting Best Practices
Tips in a Nutshell¶
- Use 4 spaces per indentation level.Never mix tabs and spaces. * Maximum Line Length should be preferably set to 79 or 80 characters * Separate top-level function and class definitions with two blank lines. * Method definitions inside a class are separated by a single blank line. * Extra blank lines may be used (sparingly) to separate groups of related functions. * Code in the core Python distribution should aways use the ASCII or Latin-1 encoding (a.k.a. ISO-8859-1). For Python 3.0 and beyond, UTF-8 is preferred over Latin-1 * Imports should usually be on separate lines rather than comma separated. * Comments that contradict the code are worse than no comments. * Comments should be complete sentences. * Block comments generally apply to some (or all) code that follows them, and are indented to the same level as that code. * Use inline comments sparingly. * Follow sections in link above for Documentation Strings, Version Bookkeeping, Naming Conventions * Avoid extraneous whitespace in the following situations:
Code Documentation Guidelines¶
Try to document your code so we can easily generate API documentation using epydoc
Refer to http://epydoc.sourceforge.net/epytext.html for EpyText? documentation
You can refer to a simple example below. Refer to epydoc example for details
def x_intercept(m, b):
"""
Return the x intercept of the line M{y=m*x+b}. The X{x intercept}
of a line is the point at which it crosses the x axis (M{y=0}).
This function can be used in conjuction with L{z_transform} to
find an arbitrary function's zeros.
@type m: number
@param m: The slope of the line.
@type b: number
@param b: The y intercept of the line. The X{y intercept} of a
line is the point at which it crosses the y axis (M{x=0}).
@rtype: number
@return: the x intercept of the line M{y=m*x+b}.
"""
return -b/m
Making a Glideinwms Release¶
Current WEB Area¶
/afs/fnal.gov/files/expwww/uscms/html/SoftwareComputing/Grid/WMS/glideinWMS
Instructions for Releasing v2_5 & Later¶
Starting with glideinWMS v2_5, glideinWMS releases are managed by gwms_release_manager package.
Release manager package is in https://cdcvs.fnal.gov/redmine/projects/gwms-release-manager
gwms_release_manager automates following tasks for you
- Creates the required checksum files in glideinwms/etc directory * Strips off the non-essential files and creates 3 release tarballs (factory, frontend and full glideinWMS release). * Anything you want to be automated during the release process should go in the release manager
Gaining Access to the Release Manager Project¶
- Register with Fermi Redmine
- Check Parag or Burt to get access to the release manager code repository
- Add following to your ~/.ssh/config
Host cdcvs.fnal.gov ForwardX11 = no GSSAPIAuthentication yes GSSAPIDelegateCredentials yes
- Get a Kerberos principle (kinit)
- Checkout the release manager code
svn checkout svn+ssh://p-gwms-release-manager@cdcvs.fnal.gov/cvs/projects/gwms-release-manager
Making a Glideinwms Release¶
- Coordinate with Mats for Corral frontend release as we also release corral frontend with glideinwms release
* Make the changes and test them.
* Update the tags file with the release notes and release version (glideinWMS/doc/tags.txt)
* Update the documentation pages glideinWMS/doc/download.html and glideinWMS/doc/history.html including possible links to be release tarballs
* Create the checksum files and release tarballs for your release. Run the command below giving the release version, --source-dir=<path to glideinwms code> and --release-dir=<path to create release related files>. Release tarballs will be created in <path to create release related files>/v2_5_1/
# For example for release v2_5_1 run the command below gwms-release-manager/trunk/bin/make_release.py \ --release-version=v2_5_1 \ --source-dir=/tmp/release/glideinWMS \ --release-dir=/tmp/release/distro
* Commit the changes to git including the changes to checksum files in etc directory * Tag (annotated) the release in git * Copy the release tarballs to the web areaFinal Releases: OLD AFS Project space: /afs/fnal.gov/files/expwww/uscms/html/SoftwareComputing/Grid/WMS/glideinWMS hosted by http://www.uscms.org/SoftwareComputing/Grid/WMS/glideinWMS # Check https://cd-docdb.fnal.gov:440/cgi-bin/ShowDocument?docid=5374 for Central Web Server Hosting Central Web Server Hosting Project Space: scp files into fnalu.fnal.gov:/web/sites/glideinwms.fnal.gov/htdocs/ Release Candidates: In 'rc' directory under the above two directories listed under final releases
* Check out the doc subdirectory in the web area. Skip for RC. * Create the required symlinks to latest docs directory. Skip for RC. * Create OSG RPMs * Announce the release to following mailing lists: glideinwms-stakeholders, cms-dct-wms@fnal.gov, glideinwms-support@fnal.gov * If you made any changes to the release manager to accommodate the release, commit those changes. * Tag the gwms_release_manager with same tag name as the glideinWMS release. * On the Redmine Project Page:
- Close all the issues related to the release. If any issues not resolved/closed assign them to future release. * Close the Version and set the appropriate release date. (Settings -> Version -> Edit the version) * Create a new version as required * Update the Custom queries to use next release version
Instructions for Releasing v2_4_x & Earlier¶
- Make the changes and test them. * Update the tags file with the release notes and release version (glideinWMS/doc/tags.txt) * Commit the changes to CVS. * Tag the release in CVS * Create and put the glideinWMS_<version>.tar.gz file in the web area * Check out the doc subdirectory in the web area. * Create the required symlinks to latest docs directory. * Update the download page and the history page with the required release information. * Announce the release to following mailing lists: osg-wms@opensciencegrid.org, cms-dct-wms@fnal.gov, glideinwms-support@fnal.gov
Creating the Glideinwms RPMs¶
Official OSG Release Site¶
Always verify official OSG release site for latest information
- https://www.opensciencegrid.org/bin/view/SoftwareTeam/RPMDevelopmentGuide
- https://www.opensciencegrid.org/bin/view/SoftwareTeam/OSGBuildTools
- https://www.opensciencegrid.org/bin/view/SoftwareTeam/ReleasePolicy
Getting Access to the Build System¶
Access needed (contact: cat@cs.wisc.edu or osg-software@opensciencegrid.org):- library.cs.wisc.edu: Or have a OSG software team upload for you
- svn access to vdt.cs.wisc.edu
- koji access (email osg-software)
Creating glideinwms RPM¶
Create a Tar glideinwms source and upload it to OSG area¶
# TAG=<glideinwms release tag to checkout> # USERNAME=<username on library.cs.wisc.edu to scp files for OSG builds> # Example: gwms-release-manager/trunk/bin/osg-release.sh v3_2_12 parag gwms-release-manager/trunk/bin/osg-release.sh $TAG $USERNAME
Build Glideinwms RPMS¶
Then checkout OSG svn:
svn co https://vdt.cs.wisc.edu/svn/native/
Or if you want just glideinwms:
https://vdt.cs.wisc.edu/svn/native/redhat/trunk/glideinwms/
In the above directory (redhat/trunk/glideinwms) you will find osg/glideinwms.spec.
This is an rpm spec file that you will use to build glideinwms
osg/glideinwms.spec ------------------ Update the release version and NVRs. NVR for final releases start with non zero int. Example: v3.2.12-1 NVR for RC will always start with 0 with following format: v3.2.12-0.1.rc1 Update any patches and set the version history changelog at the bottom of the upstream/developer.tarball.source ----------------------------- Update the location of tarball in upstream/developer.tarball.source and commit it.
Once you have a working build box (see https://twiki.grid.iu.edu/bin/view/SoftwareTeam/VDTRPMBuildBox)
you can use osg-build:
[root@client ~]$ yum --enablerepo=osg-development install osg-build
For example, running in the glideinwms directory locally:
osg-build mock .
Once the above works, you can build to koji and to the repositories:
osg-build koji --scratch .
This will do a scratch build that will build everything in a test environment, but will not push it into any repositories.
You will be able to see the progress at https://koji-hub.batlab.org/koji/
Once this works, you can run a real koji build:
osg-build koji .
This will build to osg-development. You will eventually need to promote to osg-testing (v2) or osg-contrib(v3).
Make sure that you verify with Brain Lin (formerly Tim Cartwright) before doing so and update
https://twiki.grid.iu.edu/bin/view/SoftwareTeam/PreReleaseNotes.
osg-promote glideinwms
(or you can run the koji commands manually):
koji tag-pkg el5-osg-testing glideinwms-2.5.5-6.osg.el5 koji tag-pkg el5-osg-contrib glideinwms-3.0.0-0pre2.osg.el5
Installing the Glideinwms RPMs¶
A short version. For more information, see:
https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallGlideinWMSFactory
https://twiki.grid.iu.edu/bin/view/Documentation/Release3/InstallGlideinWMSFrontend
Frontend¶
For v2+, use $REPO=osg-development
For v3, use $REPO=osg-contrib
yum --enablerepo=$REPO install glideinwms-vofrontend vi /etc/gwms-frontend/frontend.xml vi /etc/httpd/conf/httpd.conf [if you want to customize http to a different port] rm /etc/condor/config.d/00personal_condor.config vi /etc/condor/certs/condor_mapfile service httpd start service condor start [put a valid proxy into /tmp/vo_proxy] service gwms-frontend upgrade service gwms-frontend start
Factory¶
For v2+, use $REPO=osg-development
For v3, use $REPO=osg-contrib
yum --enablerepo=$REPO install glideinwms-vofrontend vi /etc/gwms-factory/glideinWMS.xml rm /etc/condor/config.d/00personal_condor.config vi /etc/condor/certs/condor_mapfile vi /etc/httpd/conf/httpd.conf [if you want to customize http to a different port] cd /var/lib/gwms-factory/condor [and then download condor tarball(s)] service httpd start service condor start service gwms-factory upgrade service gwms-factory start
Testing on Fermicloud¶
See the previous section to install the Frontend and Factory RPMs.
Fermicloud hosts also the following infrastructure that can help your testing:- A all-in-one Computer Element (Globus and HTCondor CE with local grid-mapfile, HTCondor slots) on fermicloud025.fnal.gov: great for small tests, makes it easy to manage new DNs
- A small computing cluster (HTCondor CE using GUMS and 4 worker nodes): worker nodes are bigger (2 cores and 8GB RAM), ideal to test partitionable slots and glidein policies
- CE is fermicloud121.fnal.gov
- Worker nodes are fermicloud111, fermicloud081, fermicloud313, fermicloud314
- A reverse web proxy for Factory and Frontend so that you can send glideins outside Fermilab (e.g. on Amazon AWS or Google Engine)
For more information refer to GlideinwmsTestInfrastructure
Glideinwms On Cloud¶
Refer GlideinwmsOnCloud
Glideinwms Accounting Improvements¶
Refer Accounting
Removing the big binaries from the tree¶
At some point, we should freeze the repository and remove the two big (100 MB) binaries from the tree -- it will speed up git clone by a factor of 10.
for x in `git branch -r | grep -v '\->' | grep origin | sed -e 's|origin/||g'`; do git co $x; done git filter-branch --index-filter 'git rm --cached --ignore-unmatch extensions/virtualmachines/additional_packages/EucaKernelModules/euca-kernel-modules-2.6.28-11* HEAD' --prune-empty --tag-name-filter cat -- --all git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d git reflog expire --expire=now --all git gc --prune=now git gc --aggressive --prune=now
Now we have a local copy that is clean.
We can then remove the old repository or force push to it.
git push newrepo --all git push newrepo --all --tags
Cleaning Remote Branches¶
# Step 1: Delete local branch git branch --delete <branch_name> # Step 2: Delete remote branch git push origin --delete <branch_name> # Step 3: Every should prune their repo to cleanup references to deleted branches git remote prune origin
Shell script for safe delete that ignores unmerged branches¶
#!/bin/sh branches="<space separated list of branches to delete>" failed="" remote_failed="" for branch in $branches do echo "DELETING LOCAL: $branch" git branch --delete $branch if [ $? -eq 0 ]; then echo "DELETING REMOTE: $branch" git push origin --delete $branch if [ $? -ne 0 ]; then echo "FAILED REMOTE DELETE: $branch" remote_failed="$remote_failed $branch" fi else echo "FAILED LOCAL DELETE: $branch" failed="$failed $branch" fi done echo "FAILED DELETE: $failed" echo "FAILED REMOTE DELETE: $remote_failed"