Project

General

Profile

Lynn's notes

hub

The mrb github commands make use of the hub command line extension.
hub documentation:

ssh github access

We use ssh github access for local repositories. You will be prompted for your passphrase 2-3 times for each repository clone. This gets old fast. See the instructions for adding your key to the ssh agent The github instructions are macOS specific.

Start the agent by hand (not recommended)

The following instructions appear to work on SL7, but must be executed for each new login session.

eval $(ssh-agent -s)
ssh-add ~/.ssh/id_rsa

If you use this method, you must explicitly clean up your ssh-agent.
killall ssh-agent

Using the ssh-init.sh script

(From Chris ssh-init.sh)

source .../ssh-init.sh 
ssh-init ~/.ssh/id_rsa
(work)
ssh-release

forks

There are two options for making a github fork. You either clone the repository and make a fork later, or make the fork at the same time as you clone the repository. In general, there is no reason to make a fork unless you have code to contribute.

  • clone and make the fork later
    mrb g --repo-type github larcore
    cd larcore
    git remote -v
    origin    git@github.com:LArSoft/larcore.git (fetch)
    origin    git@github.com:LArSoft/larcore.git (push)
    (do stuff)
    hub fork --remote-name origin
    origin    git@github.com:lgarren/larcore.git (fetch)
    origin    git@github.com:lgarren/larcore.git (push)
    upstream    git@github.com:LArSoft/larcore.git (fetch)
    upstream    git@github.com:LArSoft/larcore.git (push)
    
  • clone and make the fork at the same time
    mrb g --repo-type github --fork lardata
    cd lardata
    git remote -v
    origin    git@github.com:lgarren/lardata.git (fetch)
    origin    git@github.com:lgarren/lardata.git (push)
    upstream    git@github.com:LArSoft/lardata.git (fetch)
    upstream    git@github.com:LArSoft/lardata.git (push)
    

Note that if you have an existing fork, you must attach to that fork.

pull requests

Pull requests can be initiated from the github website or from your local repository clone.
How to do this from a local clone:

git checkout -b feature/<name>_<description>
make your changes
git push origin feature/<name>_<description>
hub pull-request -m "your comment here" 

pull request management

Email is sent when a pull request is initiated. All actions must be explicitly initiated by the managers. Managers can either reply to the email or use the messages on the website to request actions.

Steps in order:
  1. code-checks
    • This step just checks that code conforms to any format rules we enforce.
    • We are running a simple script to check for trailing whitespace.
  2. trigger build
    • This step will build the code and then run CI tests
  3. make changes if either requested or required
    • Simply push your changes to origin. That will initiate the process again.
  4. approve
    • Note that we will require signoff from an appropriate level 2 manager.
  5. merge
    • Only level 1 managers can merge
    • Do this from the github web page?
    • Double check that the merge will be to develop
    • You will find a merge button near the bottom of the messages.
  6. DO NOT merge if this is a breaking change
    • In this case, the merge will be handled along with a release build.

get a local copy of a pull request

cd <repository>
hub pr list
git fetch origin pull/<pull number>/head:<temporary branch name>
git checkout <temporary branch name>

Trigger a CI build with experiment feature branches

the trigger option that handles LArSoft PRs and feature branches from experiment code is --revisions,
a command example would look like:

trigger --build-delay 0 --cert ${CI_CERT} --version develop --workflow default_wf --revisions
"LArSoft/larreco#5,LArSoft/larcore#3,dunetpc@feature/bla1,uboonecode@feature/something1,ubreco@feature/something2" 

(the field separator can be either space or comma)

the options --version develop and --workflow default_wf in this example are using the default values.

To get the cert, it is enough to run the kx509 command, then the cert path can be provided to the option --cert or pointed through an environment variable.

Eventually the trigger command can include the option to select the platform (slf6, slf7, both if omitted)

--force-platform slf7

and to select qualifiers
--quals e19:py2:prof (this is the default, for the slf7 the 'py2' is automatically removed from the qualifier)

example

The certificate is generated in /tmp

$ kx509
Authorizing ...... authorized
Fetching certificate ..... fetched
Storing certificate in /tmp/x509up_u1147
Your certificate is valid until: Wed Feb 26 12:58:18 2020

Get the environment:

source /cvmfs/fermilab.opensciencegrid.org/products/common/etc/setups.sh
setup lar_ci

Trigger a build:

$ trigger --build-delay 0 --cert /tmp/x509up_u1147 --version develop --workflow default_wf --revisions "LArSoft/larsoft#8 LArSoft/larsim#6 dunetpc@feature/lg_LegacyLArG4" 
Checking if /tmp/x509up_u1147 can be reused ... yes
Triggering CI build
workflow: default_wf
trigger attempt # 1/10
trigger exit status: 0 
trigger of CI build succeeded

making a release

Releases will be made by level 1 managers.
Although the level 1 managers have permissions to submit code directly to github, we request that all changes be made via pull request.

Setup the working environment

source /cvmfs/larsoft.opensciencegrid.org/products/setup
export PRODUCTS=$PRODUCTS:/cvmfs/sbnd.opensciencegrid.org/products/sbnd:/cvmfs/dune.opensciencegrid.org/products/dune:/cvmfs/uboone.opensciencegrid.org/products:/cvmfs/icarus.opensciencegrid.org/products/icarus:/cvmfs/lariat.opensciencegrid.org/externals
setup mrb
setup larreltools
export PATH=$LARRELTOOLS_DIR/expert:${PATH}
export MRB_PROJECT=larsoft

Dependency order

  • Within the larsoftobj suite
    • larcoreobj
    • larcorealg
    • lardataobj
    • lardataalg
    • larsoftobj
  • All larsoft packages
    • larpandoracontent (depends only on pandora)
    • larcoreobj
    • larcorealg
    • larcore
    • lardataobj
    • lardataalg
    • larsoftobj
    • lardata
    • larevt
    • larwirecell (depends on larevt)
    • larg4 (depends on larevt)
    • larsim (depends on larevt)
    • larreco
    • larana (depends on larreco)
    • larpandora (depends on larreco)
    • lareventdisplay (depends on larreco)
    • larexamples (depends on larreco)
    • larsoft

First simple test

For this test, a pull request was made and merged in lardata. There are no unmerged pull requests.

mkdir some_working_dir
cd some_working_dir
mkdir e17p
mrb newDev -v v08_36_01 -q e17:prof -f -T e17p
source e17p/localProducts_larsoft_v08_36_01_e17_prof/setup
cd $MRB_SOURCE
mrb g --repo-type github --fork larsoft_suite
mrb g --repo-type github larsoftobj_suite
(no changes expected in the larsoftobj suite, so not making a fork)
tagLAr start v08_36_01
updateVersion micro
mkdir ../notag

Now step through the repositories in build/dependency order starting with larsoftobj.
cd <repository>
git diff LARSOFT_SUITE_v08_36_00 
IF there are no changes, restore the old release number with mrb uv
For example:
mrb uv larcoreobj v08_07_00
cd ..
mv <repository> ../notag
Once a repository with changes (lardata) has been found, the remaining repositories will have updated versions.

When you have finished checking the repositories, it is time for a test build.
cd $MRB_SOURCE
mrb uc
cd larsoft 
git commit -m"product versions" releaseDB/CMakeLists.txt
(make a similar commit in larsoftobj if it has changed)

We need to commit changes and push the release branch to our fork.
cd $MRB_SOURCE
makeReleasePR v08_36_01

After a pull request is made, the code checks will be run automatically. Other steps must be initiated separately.
Use the "hub pr list" command in each repository to check status from the command line.

Developing some scripts

Establish the working environment as before (setup mrb, larreltools, etc.).
Use makeLArWorkDir to create a working directory and populate $MRB_SOURCE from github.

makeLArWorkDir `pwd` v08_37_00
cd v08_37_00
source e17p/localProducts_larsoft_v08_36_02_e17_prof/setup 
cd $MRB_SOURCE

Now use manageLArGithub
Usage: manageLArGithub <start|update|commit|pr> <tag>
   start - create release/<tag> branch
   update - increment version number in product_deps
   commit - commit changes in product_deps and push the fork
   pr - initiate a pull request

We can use the mrb build and test options.
Nonbreaking PRs will already be tested and merged with develop.
However, there may be breaking PRs that must be merged with the pull request at this step.
We will continue to use the same protocol used when building releases from redmine repositories.
manageLArGithub start v08_37_00

This step will create a release branch in each local repository.
Merge in any outstanding PR's at this time.
Decide which packages will need new releases using the dependency list above.
Move packages that will not need to be updated out of the way.
manageLArGithub update

At this point, you should have a consistent set of packages with updates in ups/product_deps.
manageLArGithub update changes the micro version, e.g., v08_36_02 will become v08_36_03.
Determine if any package needs to have the minor version updated instead and use mrb uv to make the change.
mrb uv larreco v08_24_00
mrb uv larsoft v08_37_00

The larsoft package will also have changes in larsoft/releaseDB.
If the larsoftobj package is being changed, it will have changes in larsoftobj/bundle.

Once the PR's are created, use buildPR to test them.
Note that the standard lar_ci will, by design, ignore parent versions in ups/product_deps.