Apache S4
  1. Apache S4
  2. S4-35

define development workflow for git

    Details

    • Type: Task Task
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: generic
    • Fix Version/s: generic
    • Labels:
      None

      Description

      We just got accepted to the ASF git program!

      We'll get a git repository hosted at Apache soon, therefore we should define a development process that is compatible with the Apache way and takes advantage of git.

      Here is a proposal, please feel free to amend/improve/reject it.

      It is inspired by the linux kernel approach, where the "benevolent dictator" is actually the S4 community, (though only committers have with write access to the blessed repository), and where contributors submit patches from their feature branches, created after rebasing on top of the latest changes from the blessed repository.

      Infrastructure:

      • Apache S4 git repository is the "blessed" repository.
      • Only S4 committers have write access.
      • Apache S4 git repository can be cloned by anyone, therefore anyone can contribute

      Repository structure:

      we could adapt a suggestion from there http://nvie.com/posts/a-successful-git-branching-model/

      • master branch holds the released code and a tag is associated to each release
      • dev branch holds the code that has been accepted for inclusion and that will be part of the next release

      Workflow:

      1. Whenever you make some changes to the codebase, it's good to have a related issue filed in the issue tracker of the project and to use a similarly named branch in your Git repository. For example, to create a branch for an issue with the key S4-42 (see http://www.apache.org/dev/git.html#workflow)
      2. you can share your code during the development of the feature by pushing it to their public repository (not sure where that will be though). For instance, one may take the github mirror of the Apache S4 repo, create an S4-42 branch, and share it.
      3. once the feature looks ready, you rebase on top of the changes from dev , generate a patch and upload it to the corresponding Jira issue. (git format-patch)
      4. people review the patch and vote on it (see decision making http://www.apache.org/foundation/how-it-works.html)
      5. when the patch is accepted, a committer commits the patch to the Apache S4 git repository (to the "dev" branch) (git am)
      6. if you work on a different feature, you simply fetch and merge the updates to "dev".

        Issue Links

          Activity

          Hide
          Matthieu Morel added a comment -

          I think it's more an IP issue: you have to grant ownership to Apache. Currently the only option, AFAIK, is to upload a patch and explicitly grant ownership to Apache (unless you are a committer).

          Show
          Matthieu Morel added a comment - I think it's more an IP issue: you have to grant ownership to Apache. Currently the only option, AFAIK, is to upload a patch and explicitly grant ownership to Apache (unless you are a committer).
          Hide
          Karthik Kambatla (Inactive) added a comment -

          Just wanted to follow up on this issue with some feedback.

          The workflow seems to be working very well, except the issue with multi-commit patches. From what I have read online, it seems like git format-patch is the right/only way to generate patches should we choose to preserve commit information. However, git format-patch creates multiple files for multiple commits, which can be cumbersome. To add to that, if the patches have issues, going back and forth and uploading multiple patch files each time leads to more issues.

          Given that we have a github account as well, what do you think of going back to the github way of collaborating? Each JIRA can be a branch and contributors can issue pull requests. Reviewing will also become significantly easier.

          Show
          Karthik Kambatla (Inactive) added a comment - Just wanted to follow up on this issue with some feedback. The workflow seems to be working very well, except the issue with multi-commit patches. From what I have read online, it seems like git format-patch is the right/only way to generate patches should we choose to preserve commit information. However, git format-patch creates multiple files for multiple commits, which can be cumbersome. To add to that, if the patches have issues, going back and forth and uploading multiple patch files each time leads to more issues. Given that we have a github account as well, what do you think of going back to the github way of collaborating? Each JIRA can be a branch and contributors can issue pull requests. Reviewing will also become significantly easier.
          Hide
          Patrick Hunt added a comment -

          not intended to be rigid and eternal but rather to keep improving as we gain experience

          that's an excellent point.

          patch reviews: do they need to be performed by committers only?

          Anyone can review patches, some projects looks for both patch contributions as well as reviews as part of granting committership. However the person who commits the code to svn is responsible for verifying the files/changes they commit. See this http://www.apache.org/dev/committers.html#committer-responsibilities

          "Applying patches: In order to grow and maintain healthy communities, committers need to discuss, review and apply patches submitted by volunteers. The Committers are also responsible for the quality and IP clearance of the code that goes into ASF repositories."

          Show
          Patrick Hunt added a comment - not intended to be rigid and eternal but rather to keep improving as we gain experience that's an excellent point. patch reviews: do they need to be performed by committers only? Anyone can review patches, some projects looks for both patch contributions as well as reviews as part of granting committership. However the person who commits the code to svn is responsible for verifying the files/changes they commit. See this http://www.apache.org/dev/committers.html#committer-responsibilities "Applying patches: In order to grow and maintain healthy communities, committers need to discuss, review and apply patches submitted by volunteers. The Committers are also responsible for the quality and IP clearance of the code that goes into ASF repositories."
          Hide
          Matthieu Morel added a comment -

          Thanks for the comments!

          I took them into account for a new version of the proposal (which by the way, is not intended to be rigid and eternal but rather to keep improving as we gain experience).

          I also added some open questions at the end.


          We'll get a git repository hosted at Apache soon, therefore we should define a development process that is compatible with the Apache way and takes advantage of git.

          Here is a proposal, please feel free to amend/improve/reject it.

          It is inspired by the linux kernel approach, where the "benevolent dictator" is actually the S4 community, (though only committers have with write access to the blessed repository), and where contributors submit patches from their feature branches, created after rebasing on top of the latest changes from the blessed repository.

          Infrastructure:

          • Apache S4 git repository is the "blessed" repository.
          • Only S4 committers have write access.
          • Apache S4 git repository can be cloned by anyone, therefore anyone can contribute

          Repository structure:

          we could adapt a suggestion from there http://nvie.com/posts/a-successful-git-branching-model/

          • master branch holds the released code and a tag is associated to each release
          • dev branch holds the code that has been accepted for inclusion and that will be part of the next release

          Workflow:

          1. You may participate to the technical discussions and submit patches, without preconditions, hence becoming a contributor (see http://www.apache.org/foundation/how-it-works.html#users )
          2. Whenever you make some changes to the codebase, it's good to have a related issue filed in the issue tracker of the project and to use a similarly named branch in your Git repository. For example, to create a branch for an issue with the key S4-42 (see http://www.apache.org/dev/git.html#workflow)
            • you can share your code during the development of the feature by pushing it to your public repository (not sure where that will be though). For instance, you may clone the github mirror of the Apache S4 repo, create an S4-42 branch, and share it.
            • it is easier to read commits if they start by the jira issue number that they are tackling (e.g. "S4-42 a description")
          3. once the feature looks ready, you rebase on top of the changes from dev , generate a patch (named by the jira number, e.g. S4-42.patch) and upload it to the corresponding Jira issue. (use "git format-patch")
            • you also need to grant authorship to the ASF during this operation
          4. people review the patch (who? not sure whether they have to be committers)
            • at the moment, there is no defined process but it will be wise to settle on a review board for complex contributions
          5. and vote on it (see decision making http://www.apache.org/foundation/how-it-works.html)
          6. when the patch is accepted, a committer commits the patch to the Apache S4 git repository (to the "dev" branch) (git am)
            • in order to clearly identify commits for a feature in the git history, we may want to use the following strategy (indeed, I'm not sure we can keep branch information in git patches):
              1. create a local temporary branch named by the id of the jira
              2. apply the patch on it
              3. merge to the dev branch without fast forward
              4. delete the local temporary branch
              5. commit
              6. push to dev
          7. mark the jira issue as resolved, and add a reference to the commit id (SHA-1) in the resolution
          8. if you work on a different feature, you simply fetch and merge the updates to "dev".

          Open questions:

          • many projects in Apache seem to require at least 1 upvote before integrating a patch, even if this patch is trivial and provided by a committer. Should we follow that strictly or be more flexible for trivial changes (i.e. a committer could upload a patch file, then directly +1 it and integrate it to the codebase)?
          • patch reviews: do they need to be performed by committers only?
          • patch review tool: we'll need to select one and set it up
          Show
          Matthieu Morel added a comment - Thanks for the comments! I took them into account for a new version of the proposal (which by the way, is not intended to be rigid and eternal but rather to keep improving as we gain experience). I also added some open questions at the end. We'll get a git repository hosted at Apache soon, therefore we should define a development process that is compatible with the Apache way and takes advantage of git. Here is a proposal, please feel free to amend/improve/reject it. It is inspired by the linux kernel approach, where the "benevolent dictator" is actually the S4 community, (though only committers have with write access to the blessed repository), and where contributors submit patches from their feature branches, created after rebasing on top of the latest changes from the blessed repository. Infrastructure: Apache S4 git repository is the "blessed" repository. Only S4 committers have write access. Apache S4 git repository can be cloned by anyone, therefore anyone can contribute Repository structure: we could adapt a suggestion from there http://nvie.com/posts/a-successful-git-branching-model/ master branch holds the released code and a tag is associated to each release dev branch holds the code that has been accepted for inclusion and that will be part of the next release Workflow: You may participate to the technical discussions and submit patches, without preconditions, hence becoming a contributor (see http://www.apache.org/foundation/how-it-works.html#users ) Whenever you make some changes to the codebase, it's good to have a related issue filed in the issue tracker of the project and to use a similarly named branch in your Git repository. For example, to create a branch for an issue with the key S4-42 (see http://www.apache.org/dev/git.html#workflow ) you can share your code during the development of the feature by pushing it to your public repository (not sure where that will be though). For instance, you may clone the github mirror of the Apache S4 repo, create an S4-42 branch, and share it. it is easier to read commits if they start by the jira issue number that they are tackling (e.g. " S4-42 a description") once the feature looks ready, you rebase on top of the changes from dev , generate a patch (named by the jira number, e.g. S4-42 .patch) and upload it to the corresponding Jira issue. (use "git format-patch") you also need to grant authorship to the ASF during this operation people review the patch (who? not sure whether they have to be committers) at the moment, there is no defined process but it will be wise to settle on a review board for complex contributions and vote on it (see decision making http://www.apache.org/foundation/how-it-works.html ) when the patch is accepted, a committer commits the patch to the Apache S4 git repository (to the "dev" branch) (git am) in order to clearly identify commits for a feature in the git history, we may want to use the following strategy (indeed, I'm not sure we can keep branch information in git patches): create a local temporary branch named by the id of the jira apply the patch on it merge to the dev branch without fast forward delete the local temporary branch commit push to dev mark the jira issue as resolved , and add a reference to the commit id (SHA-1) in the resolution if you work on a different feature, you simply fetch and merge the updates to "dev". Open questions: many projects in Apache seem to require at least 1 upvote before integrating a patch, even if this patch is trivial and provided by a committer. Should we follow that strictly or be more flexible for trivial changes (i.e. a committer could upload a patch file, then directly +1 it and integrate it to the codebase)? patch reviews: do they need to be performed by committers only? patch review tool: we'll need to select one and set it up
          Hide
          Patrick Hunt added a comment -

          looks good to me. What are your plans re review? Free form or some suggested collaboration tool? There's http://reviews.apache.org however some projects have switch to gerrit or phabricator (see HIVE-2486)

          Show
          Patrick Hunt added a comment - looks good to me. What are your plans re review? Free form or some suggested collaboration tool? There's http://reviews.apache.org however some projects have switch to gerrit or phabricator (see HIVE-2486 )
          Hide
          Leo Neumeyer added a comment -

          Looks good, thank you for putting this together!

          There are different processes and responsibilities for contributors and committers, right? Can you clarify that?

          Show
          Leo Neumeyer added a comment - Looks good, thank you for putting this together! There are different processes and responsibilities for contributors and committers, right? Can you clarify that?
          Hide
          Karthik Kambatla (Inactive) added a comment -

          +1

          At the committers end, it might be nice to create a branch S4-42, apply the patch, and then merge with no-fastforward to keep track of each JIRA addressed.

          Show
          Karthik Kambatla (Inactive) added a comment - +1 At the committers end, it might be nice to create a branch S4-42 , apply the patch, and then merge with no-fastforward to keep track of each JIRA addressed.

            People

            • Assignee:
              Unassigned
              Reporter:
              Matthieu Morel
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:

                Development