Uploaded image for project: 'Bigtop'
  1. Bigtop
  2. BIGTOP-1429

Apache build of Bigtop test artifacts fails to find correct Hbase jars

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.7.0
    • Fix Version/s: 0.8.0
    • Component/s: build, hbase, tests
    • Labels:
      None

      Description

      Our testartifacts build fails on Apache CI system with the following error:

      [ERROR] Failed to execute goal on project hbase-smoke: Could not resolve dependencies for project org.apache.bigtop.itest:hbase-smoke:jar:0.8.0-SNAPSHOT: The following artifacts could not be resolved: org.apache.hbase:hbase-common:jar:0.98.2, org.apache.hbase:hbase-common:jar:tests:0.98.2, org.apache.hbase:hbase-server:jar:0.98.2, org.apache.hbase:hbase-server:jar:tests:0.98.2: Could not find artifact org.apache.hbase:hbase-common:jar:0.98.2 in central (http://repo.maven.apache.org/maven2) -> [Help 1]
      

      Here's the problem:

      • our packages are created on a separate Bigtop CI system that creates and locally installs HBase artifacts built against a particular Hadoop2 version. The resulting artifacts version is - in the current BOM - 0.98.2.
      • the testartifacts build in question is ran on a Apache CI machine, as it needs to publish Bigtop test artifacts to ASF maven repo (Bigtop CI machine doesn't have correct credentials for the deployment, e.g. it isn't allowed to)
      • ASF CI machine doesn't have HBase 0.98.2 artifact locally installed, and tries to fetch them from repo.maven.apache.org. However, HBase releases are produced for both hadoop1 and hadoop2. To make this distinction the artifacts versions are alternated to 0.98.2-hadoop1, and 0.98.2-hadoop2 respectively. As the result, testartifacts build fails being unable to resolve the depedencies.

      There're a few ways to solve the issue:

      1. provision ASF CI hosts beforehand with .m2 cache from bigtop CI machine
      2. deploy artifacts from Bigtop package build phase into 3rd party Maven repo, which can be later used in testartifacts build to resolve the dependencies
      3. finally, alternate Bigtop Hbase package build to produce 0.98.2-hadoop2 artifact. This version will be resolved during testartifacts and the deploy will happen. However, during the cluster tests (at Bigtop CI slaves) correct version from local .m2 caches will be used, hence guaranteeing consistency of the stack

        Issue Links

          Activity

          Hide
          cos Konstantin Boudnik added a comment -

          As much as I hate this, here's the patch for #3 solution. I pretty much agree upfront that this is crap, but at least it allows us to work around the problem without heavy dependencies on additional external crutches.

          Show
          cos Konstantin Boudnik added a comment - As much as I hate this, here's the patch for #3 solution. I pretty much agree upfront that this is crap, but at least it allows us to work around the problem without heavy dependencies on additional external crutches.
          Hide
          apurtell Andrew Purtell added a comment - - edited
          Show
          apurtell Andrew Purtell added a comment - - edited Yuck. See also https://issues.apache.org/jira/browse/BIGTOP-1427?focusedCommentId=14123329&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14123329 where this is a problem internal to the Bigtop build already (apparently)
          Hide
          apurtell Andrew Purtell added a comment -

          Reluctant +1 given both this issue and BIGTOP-1427, but HBASE_VERSION in the bomfile should be updated with the -hadoop2 suffix rather than hack it in the do-component-build scripts.

          Show
          apurtell Andrew Purtell added a comment - Reluctant +1 given both this issue and BIGTOP-1427 , but HBASE_VERSION in the bomfile should be updated with the -hadoop2 suffix rather than hack it in the do-component-build scripts.
          Hide
          plinnell Peter Linnell added a comment -

          +1 I do not see another easy fix,

          Show
          plinnell Peter Linnell added a comment - +1 I do not see another easy fix,
          Hide
          cos Konstantin Boudnik added a comment -

          Andrew Purtell, thanks for pointing to the other ticket - I missed it. I hate this "fix" as well. Please clarify how the version of a component from a BOM file will make it into hardcoded version of HBase it the corresponding pom.xml file? If you see a better way of fixing it - I am all for it!

          Show
          cos Konstantin Boudnik added a comment - Andrew Purtell , thanks for pointing to the other ticket - I missed it. I hate this "fix" as well. Please clarify how the version of a component from a BOM file will make it into hardcoded version of HBase it the corresponding pom.xml file? If you see a better way of fixing it - I am all for it!
          Hide
          cos Konstantin Boudnik added a comment -

          +1 I do not see another easy fix,

          that is until we switch to HBase 1.0 which won't be doing Hadoop1 support as Andrew Purtell has pointed out elsewhere.

          Show
          cos Konstantin Boudnik added a comment - +1 I do not see another easy fix, that is until we switch to HBase 1.0 which won't be doing Hadoop1 support as Andrew Purtell has pointed out elsewhere.
          Hide
          cos Konstantin Boudnik added a comment -

          Committed as 731bb44

          Show
          cos Konstantin Boudnik added a comment - Committed as 731bb44

            People

            • Assignee:
              cos Konstantin Boudnik
              Reporter:
              cos Konstantin Boudnik
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development