Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1363

(Discussion) Bigtop may manage apache patches

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: build
    • Labels:
      None

      Description

      Bigtop may manage apache patches.

      Motivation 1: component build fails due to some apache patches. bigtop don't need to wait for component rebase to workaround component build failure.

      Example 1) Hadoop build fails in bigtop-0.6.0/bigtop-0.7.0 build fails due to HADOOP-10110.patch missing.
      Example 2) flume 1.4.0 build fails due to Flume-2172

      Motivation 2: third party can use bigtop to manage apache or vendor patches (Not bigtop goal, but if more people use bigtop to do distribution, bigtop is more stable)

        Issue Links

          Activity

          Hide
          oflebbe Olaf Flebbe added a comment -

          Accidently stumbled over this old one.

          Hi I was not aware of this JIRA while adding patches to bigtop. Anyway, they are already there. JIRA can be closed.

          Show
          oflebbe Olaf Flebbe added a comment - Accidently stumbled over this old one. Hi I was not aware of this JIRA while adding patches to bigtop. Anyway, they are already there. JIRA can be closed.
          Hide
          cos Konstantin Boudnik added a comment -

          Are you talking about separating packaging code (which is already separated under bigtop_packages) or something else? Sorry, I am a bit confused.

          Show
          cos Konstantin Boudnik added a comment - Are you talking about separating packaging code (which is already separated under bigtop_packages) or something else? Sorry, I am a bit confused.
          Hide
          rguo Guo Ruijing added a comment -

          Hi, Konstantin

          We should decouple source code management with build framework, right?

          bigtop directory can be (components is new directory):
          [hadoop@localhost apache]$ ls -lad bigtop-master/components/*
          drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/hadoop
          drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/hbase
          drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/hive
          drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/oozie

          Show
          rguo Guo Ruijing added a comment - Hi, Konstantin We should decouple source code management with build framework, right? bigtop directory can be (components is new directory): [hadoop@localhost apache] $ ls -lad bigtop-master/components/* drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/hadoop drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/hbase drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/hive drwxr-xr-x. 1 468604085 720748206 68 Jul 10 02:37 bigtop-master/components/oozie
          Hide
          cos Konstantin Boudnik added a comment -

          Using repo is an option, but I am afraid it will complicate the build logic quite a bit, won't it?
          BTW, just a reminder: we have make-based and gradle-based package build systems nowadays.

          Show
          cos Konstantin Boudnik added a comment - Using repo is an option, but I am afraid it will complicate the build logic quite a bit, won't it? BTW, just a reminder: we have make-based and gradle-based package build systems nowadays.
          Hide
          rguo Guo Ruijing added a comment -

          +1 on Sean's proposal.

          Bigtop may use repo to manage source code branches

          a. repo: https://source.android.com/source/using-repo.html

          b. default.xml example:

          <?xml version="1.0" encoding="UTF-8"?>
          <manifest>
          <remote name="origin"
          fetch="https://github.com/apache />
          <default revision="master"
          remote="origin"/>
          <project path="zookeeper" name="zookeeper.git" groups="stack" revision="branch-3.4"/>
          <project path="hadoop" name="hadoop.git" groups="stack" revision="branch-2.2.0"/>
          <project path="hbase" name="hbase.git" groups="stack" revision="branch-0.96"/>
          <project path="hive" name="hive.git" groups="stack" revision="branch-0.12"/>
          <project path="pig" name="pig.git" groups="stack" revision="branch-0.12"/>
          <project path="sqoop" name="sqoop.git" groups="stack" revision="branch-1.4.2"/>
          <project path="mahout" name="mahout.git" groups="stack" revision="branch-0.7"/>
          <project path="flume" name="flume.git" groups="stack" revision="flume-1.4"/>
          <project path="oozie" name="oozie.git" groups="stack" revision="branch-4.0.0"/>
          </manifest>

          Show
          rguo Guo Ruijing added a comment - +1 on Sean's proposal. Bigtop may use repo to manage source code branches a. repo: https://source.android.com/source/using-repo.html b. default.xml example: <?xml version="1.0" encoding="UTF-8"?> <manifest> <remote name="origin" fetch="https://github.com/apache /> <default revision="master" remote="origin"/> <project path="zookeeper" name="zookeeper.git" groups="stack" revision="branch-3.4"/> <project path="hadoop" name="hadoop.git" groups="stack" revision="branch-2.2.0"/> <project path="hbase" name="hbase.git" groups="stack" revision="branch-0.96"/> <project path="hive" name="hive.git" groups="stack" revision="branch-0.12"/> <project path="pig" name="pig.git" groups="stack" revision="branch-0.12"/> <project path="sqoop" name="sqoop.git" groups="stack" revision="branch-1.4.2"/> <project path="mahout" name="mahout.git" groups="stack" revision="branch-0.7"/> <project path="flume" name="flume.git" groups="stack" revision="flume-1.4"/> <project path="oozie" name="oozie.git" groups="stack" revision="branch-4.0.0"/> </manifest>
          Hide
          rvs Roman Shaposhnik added a comment -

          +1 on Sean's proposal as well!

          Show
          rvs Roman Shaposhnik added a comment - +1 on Sean's proposal as well!
          Hide
          cos Konstantin Boudnik added a comment -

          +1 on Sean's proposal. I think we even have a half-abandoned almost complete solution for it somewhere in the JIRA. I will dry to dig it out over the weekend.

          Show
          cos Konstantin Boudnik added a comment - +1 on Sean's proposal. I think we even have a half-abandoned almost complete solution for it somewhere in the JIRA. I will dry to dig it out over the weekend.
          Hide
          mgrover Mark Grover added a comment -

          Big +1 to what Sean said. I don't think individual project release managers want to release busted releases, If we were to do a better job of helping them stabilize their releases (and Sean's suggestion opens that avenue), I think we would see a big reduction in the number of cases we get bitten by busted releases.

          I think while you still have a valid point about Bigtop managing patches, but IMO that comes at a good chunk of cost and complexity. Integrating directly with source code repositories of upstream projects may give us a much better bang for the buck.

          Show
          mgrover Mark Grover added a comment - Big +1 to what Sean said. I don't think individual project release managers want to release busted releases, If we were to do a better job of helping them stabilize their releases (and Sean's suggestion opens that avenue), I think we would see a big reduction in the number of cases we get bitten by busted releases. I think while you still have a valid point about Bigtop managing patches, but IMO that comes at a good chunk of cost and complexity. Integrating directly with source code repositories of upstream projects may give us a much better bang for the buck.
          Hide
          apurtell Andrew Purtell added a comment -

          I'd like to suggest an alternate solution to the problem that we've wanted to do for a long time anyway - make it easy to build from a branch of a source repository, rather than just published tarballs. Bigtop releases would still be entirely Apache releases, but developers could routinely build from trunk or curate their own branches.

          I am a huge fan of this idea.

          Show
          apurtell Andrew Purtell added a comment - I'd like to suggest an alternate solution to the problem that we've wanted to do for a long time anyway - make it easy to build from a branch of a source repository, rather than just published tarballs. Bigtop releases would still be entirely Apache releases, but developers could routinely build from trunk or curate their own branches. I am a huge fan of this idea.
          Hide
          mackrorysd Sean Mackrory added a comment -

          We've very occasionally bundled patches that fix build-related stuff, but yes to support this generally would very much, IMO, be a departure from existing policy.

          I'd like to suggest an alternate solution to the problem that we've wanted to do for a long time anyway - make it easy to build from a branch of a source repository, rather than just published tarballs. Bigtop releases would still be entirely Apache releases, but developers could routinely build from trunk or curate their own branches.

          Show
          mackrorysd Sean Mackrory added a comment - We've very occasionally bundled patches that fix build-related stuff, but yes to support this generally would very much, IMO, be a departure from existing policy. I'd like to suggest an alternate solution to the problem that we've wanted to do for a long time anyway - make it easy to build from a branch of a source repository, rather than just published tarballs. Bigtop releases would still be entirely Apache releases, but developers could routinely build from trunk or curate their own branches.
          Hide
          jayunit100 jay vyas added a comment -

          This is a requirements change ? https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27825348 Very clearly states that bigtop DOES NOT patch hadoop.
          So I guess first some discussion about that policy should go on mailing list.

          Show
          jayunit100 jay vyas added a comment - This is a requirements change ? https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=27825348 Very clearly states that bigtop DOES NOT patch hadoop. So I guess first some discussion about that policy should go on mailing list.
          Hide
          rguo Guo Ruijing added a comment -

          new function in package.mk:

          1. create new file in $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/patches

          2. file patches example:

          HADOOP-10110.patch p0
          HDFS-2139.patch p1

          3. read patch file and strip level from patches and patch one file by one file

          Show
          rguo Guo Ruijing added a comment - new function in package.mk: 1. create new file in $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/patches 2. file patches example: HADOOP-10110 .patch p0 HDFS-2139 .patch p1 3. read patch file and strip level from patches and patch one file by one file
          Hide
          rguo Guo Ruijing added a comment -

          solution: bigtop do different patches before build.

          existing function in package.mk:

          if [ -f $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/series ]; then \
          PATCHES="`cat $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/series`" ;\
          elif [ -f $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/patch ]; then \
          PATCHES="patch" ;\
          else \
          PATCHES="/dev/null" ;\
          fi ; (cd $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME); cat $$PATCHES)| \
          (cd $(PKG_BUILD_DIR)/tar/$($(PKG)_NAME)-$(PKG_PKG_VERSION)$(BIGTOP_BUILD_STAMP) ; patch -p0)

          Show
          rguo Guo Ruijing added a comment - solution: bigtop do different patches before build. existing function in package.mk: if [ -f $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/series ]; then \ PATCHES="`cat $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/series`" ;\ elif [ -f $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME)/patch ]; then \ PATCHES="patch" ;\ else \ PATCHES="/dev/null" ;\ fi ; (cd $(BASE_DIR)/bigtop-packages/src/common/$($(PKG)_NAME); cat $$PATCHES)| \ (cd $(PKG_BUILD_DIR)/tar/$($(PKG)_NAME)-$(PKG_PKG_VERSION)$(BIGTOP_BUILD_STAMP) ; patch -p0)

            People

            • Assignee:
              Unassigned
              Reporter:
              rguo Guo Ruijing
            • Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development