Details

    • Type: Sub-task Sub-task
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.8.0
    • Fix Version/s: 0.8.0
    • Component/s: debian, rpm
    • Labels:
      None

      Description

      See https://issues.apache.org/jira/browse/CRUNCH-386 for details.

      We need Crunch 0.10 for use with HBase 0.98+

        Issue Links

          Activity

          Hide
          Roman Shaposhnik added a comment -

          Thanks, Mark Grover! Any ETA on the patch?

          Show
          Roman Shaposhnik added a comment - Thanks, Mark Grover ! Any ETA on the patch?
          Hide
          Mark Grover added a comment -

          Sure, I haven't looked into it but I am going to aim for the end of the week if not before. Let me know if that's not ok.

          Show
          Mark Grover added a comment - Sure, I haven't looked into it but I am going to aim for the end of the week if not before. Let me know if that's not ok.
          Hide
          Roman Shaposhnik added a comment -

          Mark Grover that's totally fine and would be very much appreciated!

          Show
          Roman Shaposhnik added a comment - Mark Grover that's totally fine and would be very much appreciated!
          Hide
          Mark Grover added a comment - - edited

          The patch I have just looking at the layout differences in Crunch is fairly simple. Simple bump of versions in bigtop.mk but I am getting the following error:

          [ERROR] Failed to execute goal on project crunch-hbase: Could not resolve dependencies for project org.apache.crunch:crunch-hbase:jar:0.10.0: The following artifacts could not be resolved: org.apache.hbase:hbase-common:jar:0.98.2, org.apache.hbase:hbase-protocol:jar:0.98.2, org.apache.hbase:hbase-client:jar:0.98.2, org.apache.hbase:hbase-shell:jar:0.98.2, org.apache.hbase:hbase-server:jar:0.98.2, org.apache.hbase:hbase-hadoop2-compat:jar:0.98.2, org.apache.hbase:hbase-testing-util:jar:0.98.2: Failure to find org.apache.hbase:hbase-common:jar:0.98.2 in http://repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1]
          [ERROR] 
          [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
          [ERROR] Re-run Maven using the -X switch to enable full debug logging.
          [ERROR] 
          [ERROR] For more information about the errors and possible solutions, please read the following articles:
          [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
          [ERROR] 
          [ERROR] After correcting the problems, you can resume the build with the command
          [ERROR]   mvn <goals> -rf :crunch-hbase
          error: Bad exit status from /var/tmp/rpm-tmp.lx5G2T (%build)
          
          
          RPM build errors:
              Bad exit status from /var/tmp/rpm-tmp.lx5G2T (%build)
          make: *** [/root/bigtop/build/crunch//.rpm] Error 1
          

          I will continue poking. Cleaning up the repo and then trying.

          Show
          Mark Grover added a comment - - edited The patch I have just looking at the layout differences in Crunch is fairly simple. Simple bump of versions in bigtop.mk but I am getting the following error: [ERROR] Failed to execute goal on project crunch-hbase: Could not resolve dependencies for project org.apache.crunch:crunch-hbase:jar:0.10.0: The following artifacts could not be resolved: org.apache.hbase:hbase-common:jar:0.98.2, org.apache.hbase:hbase-protocol:jar:0.98.2, org.apache.hbase:hbase-client:jar:0.98.2, org.apache.hbase:hbase-shell:jar:0.98.2, org.apache.hbase:hbase-server:jar:0.98.2, org.apache.hbase:hbase-hadoop2-compat:jar:0.98.2, org.apache.hbase:hbase-testing-util:jar:0.98.2: Failure to find org.apache.hbase:hbase-common:jar:0.98.2 in http: //repo.maven.apache.org/maven2 was cached in the local repository, resolution will not be reattempted until the update interval of central has elapsed or updates are forced -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch . [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http: //cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn <goals> -rf :crunch-hbase error: Bad exit status from / var /tmp/rpm-tmp.lx5G2T (%build) RPM build errors: Bad exit status from / var /tmp/rpm-tmp.lx5G2T (%build) make: *** [/root/bigtop/build/crunch //.rpm] Error 1 I will continue poking. Cleaning up the repo and then trying.
          Hide
          Roman Shaposhnik added a comment -

          I suggest running hbase build first and see it that helps (since HBase artifacts will be deposited into .m2)

          Show
          Roman Shaposhnik added a comment - I suggest running hbase build first and see it that helps (since HBase artifacts will be deposited into .m2)
          Hide
          Mark Grover added a comment -

          Thanks Roman!
          I see, that's what I was thinking. But should we be using the maven artifacts generated by bigtop builds or the maven artifacts generated by the upstream builds.

          If the former, is it safe to assume that any build slave in Bigtop jenkins that will build crunch would have had an hbase build ran on it previously?

          Show
          Mark Grover added a comment - Thanks Roman! I see, that's what I was thinking. But should we be using the maven artifacts generated by bigtop builds or the maven artifacts generated by the upstream builds. If the former, is it safe to assume that any build slave in Bigtop jenkins that will build crunch would have had an hbase build ran on it previously?
          Hide
          Roman Shaposhnik added a comment -

          We, of course, should be using the artifacts generated by our own builds. That however, would require an extra bit of setup: Bigtop Maven repo. It may very well be that its high time for us to bite the bulled and do it.

          Show
          Roman Shaposhnik added a comment - We, of course, should be using the artifacts generated by our own builds. That however, would require an extra bit of setup: Bigtop Maven repo. It may very well be that its high time for us to bite the bulled and do it.
          Hide
          Mark Grover added a comment -

          Now, I am running into issues building HBase:

          [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile (default-compile) on project hbase-client: Compilation failure: Compilation failure:
          [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[158,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds Q,com.google.protobuf.Message
          [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[253,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds Q,com.google.protobuf.Message
          [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[407,55] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message
          [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[481,55] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message
          [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[587,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message
          [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[706,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message
          

          It looks Java version related and this showed up even with 1.6 45, but I will make sure it's get set properly and try again.

          Show
          Mark Grover added a comment - Now, I am running into issues building HBase: [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile ( default -compile) on project hbase-client: Compilation failure: Compilation failure: [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[158,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds Q,com.google.protobuf.Message [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[253,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds Q,com.google.protobuf.Message [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[407,55] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[481,55] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[587,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message [ERROR] /root/bigtop/build/hbase/rpm/BUILD/hbase-0.98.2/hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/AggregationClient.java:[706,57] type parameters of <T>T cannot be determined; no unique maximal instance exists for type variable T with upper bounds T,com.google.protobuf.Message It looks Java version related and this showed up even with 1.6 45, but I will make sure it's get set properly and try again.
          Hide
          Roman Shaposhnik added a comment -

          I am actually sort of happy it shows up with 45 as well since I could figure out for the life of me why it would make a difference. Andrew Purtell looks like this is still happening.

          Show
          Roman Shaposhnik added a comment - I am actually sort of happy it shows up with 45 as well since I could figure out for the life of me why it would make a difference. Andrew Purtell looks like this is still happening.
          Hide
          Andrew Purtell added a comment -

          I swear we can all build HBase 0.98 just fine with Oracle 6u45, and see the screen capture here https://issues.apache.org/jira/browse/BIGTOP-1110?focusedCommentId=14044099&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14044099 . What more can I say.

          Show
          Andrew Purtell added a comment - I swear we can all build HBase 0.98 just fine with Oracle 6u45, and see the screen capture here https://issues.apache.org/jira/browse/BIGTOP-1110?focusedCommentId=14044099&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14044099 . What more can I say.
          Hide
          Mark Grover added a comment -

          Yeah, I will take another look. Thanks Andrew and Roman for your input!

          Show
          Mark Grover added a comment - Yeah, I will take another look. Thanks Andrew and Roman for your input!
          Hide
          Roman Shaposhnik added a comment - - edited

          Btw, keep in mind that you may want to adjust the version of HBase that do-component-build passes to the Crunch build. There needs to be -hadoop2 at the end. In fact, perhaps we should do it as a whole sale change really for all of the components of Bigtop.

          E.g. here's what I had to do to fix Flume: https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blobdiff;f=bigtop-packages/src/common/flume/do-component-build;h=9a8a4c34179c78b1deb685024a8d4db5defe143d;hp=519759554c22acdabc09da873b276b1dd40c0134;hb=b39748a22fb985a1352f5e20c481d521840f5da8;hpb=30c8e69ae391ffa6569f92f9ae3da1b20e317b67

          Show
          Roman Shaposhnik added a comment - - edited Btw, keep in mind that you may want to adjust the version of HBase that do-component-build passes to the Crunch build. There needs to be -hadoop2 at the end. In fact, perhaps we should do it as a whole sale change really for all of the components of Bigtop. E.g. here's what I had to do to fix Flume: https://git-wip-us.apache.org/repos/asf?p=bigtop.git;a=blobdiff;f=bigtop-packages/src/common/flume/do-component-build;h=9a8a4c34179c78b1deb685024a8d4db5defe143d;hp=519759554c22acdabc09da873b276b1dd40c0134;hb=b39748a22fb985a1352f5e20c481d521840f5da8;hpb=30c8e69ae391ffa6569f92f9ae3da1b20e317b67
          Hide
          Mark Grover added a comment -

          Something was wrong in my environment, HBase builds now. Now, I am back to building crunch.

          And, thanks for that tip, Roman, I am adding -hadoop2 as suffix to hbase artifacts.

          Show
          Mark Grover added a comment - Something was wrong in my environment, HBase builds now. Now, I am back to building crunch. And, thanks for that tip, Roman, I am adding -hadoop2 as suffix to hbase artifacts.
          Hide
          Mark Grover added a comment -

          Can I get a review on this please? It's a blocker for 0.8.0

          Show
          Mark Grover added a comment - Can I get a review on this please? It's a blocker for 0.8.0
          Hide
          Roman Shaposhnik added a comment -

          +1. Please commit.

          Show
          Roman Shaposhnik added a comment - +1. Please commit.

            People

            • Assignee:
              Mark Grover
              Reporter:
              Mark Grover
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development