Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1427

HBase build should build hbase-***-hadoop2 by default

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: build
    • Labels:
      None

      Description

      Flume is depending on hbase-**-hadoop2, but the hbase build is not providing this.

        Issue Links

          Activity

          Hide
          apurtell Andrew Purtell added a comment -

          Upstream HBase munges the POM to produce -hadoop1 and -hadoop2 artifacts. We will be ending this practice after HBase 0.98 by dropping support for Hadoop 1.x with HBase 1.0.

          Bigtop builds HBase artifacts without this munging. I don't think Bigtop should start doing that.

          This does look like an issue with Flume's do-component-build script:

          mvn -DskipTests -Dhadoop.profile=hbase-98  \
            -Dhadoop2.version=$HADOOP_VERSION        \
            -Dhbase.version=${HBASE_VERSION}-hadoop2 \
            clean install "$@"
          

          That's only going to work if upstream put out the specified -hadoop2 artifact in Maven central. It won't correspond with the local HBase artifacts Bigtop may have built otherwise.

          Show
          apurtell Andrew Purtell added a comment - Upstream HBase munges the POM to produce -hadoop1 and -hadoop2 artifacts. We will be ending this practice after HBase 0.98 by dropping support for Hadoop 1.x with HBase 1.0. Bigtop builds HBase artifacts without this munging. I don't think Bigtop should start doing that. This does look like an issue with Flume's do-component-build script: mvn -DskipTests -Dhadoop.profile=hbase-98 \ -Dhadoop2.version=$HADOOP_VERSION \ -Dhbase.version=${HBASE_VERSION}-hadoop2 \ clean install "$@" That's only going to work if upstream put out the specified -hadoop2 artifact in Maven central. It won't correspond with the local HBase artifacts Bigtop may have built otherwise.
          Hide
          mgrover Mark Grover added a comment -

          Thanks Andrew!
          So, what do you suggest Bigtop build -hadoop2 artifacts for HBase, or instead we remove the "-hadoop2" identifier from Flume's build? I remember there was a problem with building Flume without it, so I am inclined to say the former but would to hear your opinion.

          Show
          mgrover Mark Grover added a comment - Thanks Andrew! So, what do you suggest Bigtop build -hadoop2 artifacts for HBase, or instead we remove the "-hadoop2" identifier from Flume's build? I remember there was a problem with building Flume without it, so I am inclined to say the former but would to hear your opinion.
          Hide
          apurtell Andrew Purtell added a comment -

          do you suggest Bigtop build -hadoop2 artifacts for HBase

          Five minutes ago I would have suggested not but then I found BIGTOP-1429 so, reluctantly, yes, accept Cos' patch on BIGTOP-1429, update where HBASE_VERSION is set in the bomfile to append the -hadoop2 suffix, and make sure the do-component-build scripts are not making local hacks.

          Show
          apurtell Andrew Purtell added a comment - do you suggest Bigtop build -hadoop2 artifacts for HBase Five minutes ago I would have suggested not but then I found BIGTOP-1429 so, reluctantly, yes, accept Cos' patch on BIGTOP-1429 , update where HBASE_VERSION is set in the bomfile to append the -hadoop2 suffix, and make sure the do-component-build scripts are not making local hacks.
          Hide
          cos Konstantin Boudnik added a comment -

          The way HBase build is done is that it has hardcoded version of, say, 0.98.2 in the pom.xml (like it should). But to produce *-hadoop[1,2] they have a script that append the version with a suffix and copies the alternated pom.xml into new files with .hadoop1 and .hadoop2 extensions. Then you run build using those new poms to produce the artifacts for different Hadoop lines. Look for yourself in the dev-support script. Yes, that's pretty bad, if you ask me. But if there any changes to make - they would have to be make in the upstream including re-thinking of the versioning of the hbase artifacts.

          Show
          cos Konstantin Boudnik added a comment - The way HBase build is done is that it has hardcoded version of, say, 0.98.2 in the pom.xml (like it should). But to produce *-hadoop [1,2] they have a script that append the version with a suffix and copies the alternated pom.xml into new files with .hadoop1 and .hadoop2 extensions. Then you run build using those new poms to produce the artifacts for different Hadoop lines. Look for yourself in the dev-support script. Yes, that's pretty bad, if you ask me. But if there any changes to make - they would have to be make in the upstream including re-thinking of the versioning of the hbase artifacts.
          Hide
          cos Konstantin Boudnik added a comment -

          BTW, we can't change the version of HBase in the bom file, because the version of the release source artifact is 0.98.2. If we use -hadoop2 suffix the build will stop working completely as it won't be able to find the source tarball.

          Show
          cos Konstantin Boudnik added a comment - BTW, we can't change the version of HBase in the bom file, because the version of the release source artifact is 0.98.2. If we use -hadoop2 suffix the build will stop working completely as it won't be able to find the source tarball.
          Hide
          cos Konstantin Boudnik added a comment -

          I guess what I am saying, is that with BIGTOP-1429 hack, and small update proposed here, Flume should be fine.

          Show
          cos Konstantin Boudnik added a comment - I guess what I am saying, is that with BIGTOP-1429 hack, and small update proposed here, Flume should be fine.
          Hide
          apurtell Andrew Purtell added a comment -

          Look for yourself in the dev-support script.

          I'm aware of this procedure as I run through it every time I make an HBase 0.98 release.

          But if there any changes to make - they would have to be make in the upstream including re-thinking of the versioning of the hbase artifacts.

          We have two binary-incompatible releases due to compiling against different major versions of Hadoop, and the POM munging is the only viable workaround of limitations of Maven classifiers that we have found. But if you have a better concrete solution, by all means present it on dev@hbase.

          BTW, we can't change the version of HBase in the bom file, because the version of the release source artifact is 0.98.2

          Ah, yeah, that's annoying.

          Show
          apurtell Andrew Purtell added a comment - Look for yourself in the dev-support script. I'm aware of this procedure as I run through it every time I make an HBase 0.98 release. But if there any changes to make - they would have to be make in the upstream including re-thinking of the versioning of the hbase artifacts. We have two binary-incompatible releases due to compiling against different major versions of Hadoop, and the POM munging is the only viable workaround of limitations of Maven classifiers that we have found. But if you have a better concrete solution, by all means present it on dev@hbase. BTW, we can't change the version of HBase in the bom file, because the version of the release source artifact is 0.98.2 Ah, yeah, that's annoying.
          Hide
          cos Konstantin Boudnik added a comment -

          I'm aware of this procedure as I run through it every time I make an HBase 0.98 release.

          I am pretty sure you are: put it here for the benefits of others. Sorry if my comment came out as a bashing: you know how it is with emails.

          Show
          cos Konstantin Boudnik added a comment - I'm aware of this procedure as I run through it every time I make an HBase 0.98 release. I am pretty sure you are: put it here for the benefits of others. Sorry if my comment came out as a bashing: you know how it is with emails.
          Hide
          apurtell Andrew Purtell added a comment -

          My apologies then.

          At every release when doing the POM munging I feel a little dirty.

          Show
          apurtell Andrew Purtell added a comment - My apologies then. At every release when doing the POM munging I feel a little dirty.
          Hide
          cos Konstantin Boudnik added a comment -

          Andrew Purtell - shall we close this as 'won't fix' because BIGTOP-1429 has been committed?

          Show
          cos Konstantin Boudnik added a comment - Andrew Purtell - shall we close this as 'won't fix' because BIGTOP-1429 has been committed?
          Hide
          rvs Roman Shaposhnik added a comment -

          stanley shi sorry for a bit of confusion, the good news is that we fixed it in a different JIRA. Please let us know if you see something else that needs to be done here.

          Show
          rvs Roman Shaposhnik added a comment - stanley shi sorry for a bit of confusion, the good news is that we fixed it in a different JIRA. Please let us know if you see something else that needs to be done here.
          Hide
          stanley_shi stanley shi added a comment -
          Show
          stanley_shi stanley shi added a comment - Thanks Roman Shaposhnik !

            People

            • Assignee:
              rvs Roman Shaposhnik
              Reporter:
              stanley_shi stanley shi
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development