Pig
  1. Pig
  2. PIG-2891

HBaseStorage can't work with HBase 0.94, as the connect method changed

    Details

      Description

      Pig HBaseStorage doesn't work against a local mode HBase 0.94.1.

      Updating the JAR in ivy/libraries.properties addresses the issue, which is pasted here:

      2012-08-24 23:27:07,431 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. Not a host:port pair: ?'58971@russell-jurneys-macbook-pro.local10.0.1.11,55171,1345873780529
      2012-08-24 23:27:07,431 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.IllegalArgumentException: Not a host:port pair: ?'58971@russell-jurneys-macbook-pro.local10.0.1.11,55171,1345873780529
      at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:60)
      at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82)
      at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:575)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:684)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:586)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:684)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
      at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
      at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
      at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
      at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:196)
      at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:593)
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
      at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
      at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
      at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
      at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
      at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
      at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
      at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
      at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
      at org.apache.pig.PigServer.compilePp(PigServer.java:1310)
      at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1247)
      at org.apache.pig.PigServer.execute(PigServer.java:1239)
      at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
      at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:136)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:197)
      at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
      at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
      at org.apache.pig.Main.run(Main.java:602)
      at org.apache.pig.Main.main(Main.java:154)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:156)

      1. PIG-2891.patch
        0.4 kB
        Russell Jurney

        Issue Links

          Activity

          Hide
          Cheolsoo Park added a comment -

          Hi Ionut,

          Please let me summarize what the current issues are with using pig 0.10.0 and hbase 0.94. Hopefully, this will help you to understand better.

          1) Pig.jar embeds hbase-0.90; therefore, if you use it with any upper versions of hbase, you will get an incompatibility exception. We could upgrade the hbase dependency in pig.jar to 0.94; however, that will cause the same problem to people who are using hbase-90. In addition, that breaks 2 unit tests (PIG-2885).

          2) The suggested workaround is using pig-withouthadoop.jar. Since pig-withouthadoop.jar doesn't contain hadoop and hbase, you have to install them and set HADOOP_HOME and HBASE_HOME. The current pig-wihtouthadoop.jar works fine with habse-0.90, 0.92, and 0.94. In fact, there was a question on the user mailing list regarding how to get pig 0.10.0 working with hbase-0.94:

          http://mail-archives.apache.org/mod_mbox/pig-user/201209.mbox/%3CCAO8ATY13+CxCZ8fkS_JSGH_Fwyxh3zB_Sxjv7iX-og5YeQi-MQ@mail.gmail.com%3E

          That being said, I don't see why this is a blocking issue. Is there any reason why you cannot use pig-withouthadoop.jar?

          Thanks!

          Show
          Cheolsoo Park added a comment - Hi Ionut, Please let me summarize what the current issues are with using pig 0.10.0 and hbase 0.94. Hopefully, this will help you to understand better. 1) Pig.jar embeds hbase-0.90; therefore, if you use it with any upper versions of hbase, you will get an incompatibility exception. We could upgrade the hbase dependency in pig.jar to 0.94; however, that will cause the same problem to people who are using hbase-90. In addition, that breaks 2 unit tests ( PIG-2885 ). 2) The suggested workaround is using pig-withouthadoop.jar. Since pig-withouthadoop.jar doesn't contain hadoop and hbase, you have to install them and set HADOOP_HOME and HBASE_HOME. The current pig-wihtouthadoop.jar works fine with habse-0.90, 0.92, and 0.94. In fact, there was a question on the user mailing list regarding how to get pig 0.10.0 working with hbase-0.94: http://mail-archives.apache.org/mod_mbox/pig-user/201209.mbox/%3CCAO8ATY13+CxCZ8fkS_JSGH_Fwyxh3zB_Sxjv7iX-og5YeQi-MQ@mail.gmail.com%3E That being said, I don't see why this is a blocking issue. Is there any reason why you cannot use pig-withouthadoop.jar? Thanks!
          Hide
          Bill Graham added a comment -

          What exactly are you blocked on? Can you not use pig-withouthadoop.jar with HBase 0.94 as Cheolsoo suggests? Please be more specific and file a JIRA if one doesn't already exist for the problem you're facing.

          If your timeline is urgent you can apply the relevant patches yourself as needed, or contribute the patch if it doesn't yet exist.

          Show
          Bill Graham added a comment - What exactly are you blocked on? Can you not use pig-withouthadoop.jar with HBase 0.94 as Cheolsoo suggests? Please be more specific and file a JIRA if one doesn't already exist for the problem you're facing. If your timeline is urgent you can apply the relevant patches yourself as needed, or contribute the patch if it doesn't yet exist.
          Hide
          Ionut Ignatescu added a comment -

          'Coming months' is too much.
          Since this is a blocking issue, I think it's neccessary a solution as soon as possible.

          Show
          Ionut Ignatescu added a comment - 'Coming months' is too much. Since this is a blocking issue, I think it's neccessary a solution as soon as possible.
          Hide
          Bill Graham added a comment -

          Discussions started this week about a Pig 0.11 branch so we could have a release in the coming months.

          Show
          Bill Graham added a comment - Discussions started this week about a Pig 0.11 branch so we could have a release in the coming months.
          Hide
          Ionut Ignatescu added a comment -

          Is there any ETA regarding Apache Pig 0.11 release date?

          Show
          Ionut Ignatescu added a comment - Is there any ETA regarding Apache Pig 0.11 release date?
          Bill Graham made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Resolution Won't Fix [ 2 ]
          Hide
          Bill Graham added a comment -

          Agreed, thanks Cheolsoo. Closing this issue.

          Show
          Bill Graham added a comment - Agreed, thanks Cheolsoo. Closing this issue.
          Hide
          Cheolsoo Park added a comment -

          Hi Bill, thanks for the question.

          So the original problem that Russell ran into is because he used the fat "pig.jar" with hbase-0.94. Since there is a mismatch between client (hbase-0.90) and server (hbase-0.94), the above exception is thrown. However, this problem can be worked around by using "pig-withouthadoop.jar", so there is no need for any immediate fix. So I think that this jira can be closed.

          In the long term, we probably want to stop embedding hbase in the fat jar as Dmitriy suggested, but that can be done part of PIG-2885. Please let me know if anyone has objections.

          Thanks!

          Show
          Cheolsoo Park added a comment - Hi Bill, thanks for the question. So the original problem that Russell ran into is because he used the fat "pig.jar" with hbase-0.94. Since there is a mismatch between client (hbase-0.90) and server (hbase-0.94), the above exception is thrown. However, this problem can be worked around by using "pig-withouthadoop.jar", so there is no need for any immediate fix. So I think that this jira can be closed. In the long term, we probably want to stop embedding hbase in the fat jar as Dmitriy suggested, but that can be done part of PIG-2885 . Please let me know if anyone has objections. Thanks!
          Bill Graham made changes -
          Patch Info Patch Available [ 10042 ]
          Hide
          Bill Graham added a comment -

          Canceling this patch since unit tests fail when upgrading, per PIG-2885. Can this be closed as a dup of PIG-2885 or are the two different issues?

          Show
          Bill Graham added a comment - Canceling this patch since unit tests fail when upgrading, per PIG-2885 . Can this be closed as a dup of PIG-2885 or are the two different issues?
          Dmitriy V. Ryaboy made changes -
          Priority Blocker [ 1 ] Major [ 3 ]
          Hide
          Dmitriy V. Ryaboy added a comment -

          removing 'blocker' priority since pig 0.11 release is by no means tied with hbase 0.94 compatibility.

          Show
          Dmitriy V. Ryaboy added a comment - removing 'blocker' priority since pig 0.11 release is by no means tied with hbase 0.94 compatibility.
          Hide
          Cheolsoo Park added a comment -

          Bumping hbase to 0.94 also breaks two unit tests: TestHbaseStorage and TestJobSubmission (PIG-2885). I was able to fix them with hadoop 1.0, but TestHbaseStorage doesn't yet pass with hadoop 0.23.1.

          Show
          Cheolsoo Park added a comment - Bumping hbase to 0.94 also breaks two unit tests: TestHbaseStorage and TestJobSubmission ( PIG-2885 ). I was able to fix them with hadoop 1.0, but TestHbaseStorage doesn't yet pass with hadoop 0.23.1.
          Cheolsoo Park made changes -
          Link This issue relates to PIG-2885 [ PIG-2885 ]
          Hide
          Dmitriy V. Ryaboy added a comment -

          As evidenced by the nature of the patch, HBaseStorage totally works with 0.94. You just need to actually use HBase 0.94 jar.

          Don't think bumping dependency is the solution, as this screws everyone using 90 and 92 just as much as users of 94 are screwed now – one has to use the core jar, and set up his own classpath, instead of using the fatjar that is mislabeled as pig-withoutdependencies (but bundles hbase, guava, jackson, and a bunch of other stuff).

          Long-term solution is probably more along the lines of no longer bundling dependencies, and making bin/pig set up the classpath for you out of core jars, respecting $HBASE_CLASSPATH and so on.

          I would also like to see the hbase loader broken out into a separate module.. that might have to wait for mavenization.

          What's 0.94's penetration these days, does anyone know?

          Show
          Dmitriy V. Ryaboy added a comment - As evidenced by the nature of the patch, HBaseStorage totally works with 0.94. You just need to actually use HBase 0.94 jar. Don't think bumping dependency is the solution, as this screws everyone using 90 and 92 just as much as users of 94 are screwed now – one has to use the core jar, and set up his own classpath, instead of using the fatjar that is mislabeled as pig-withoutdependencies (but bundles hbase, guava, jackson, and a bunch of other stuff). Long-term solution is probably more along the lines of no longer bundling dependencies, and making bin/pig set up the classpath for you out of core jars, respecting $HBASE_CLASSPATH and so on. I would also like to see the hbase loader broken out into a separate module.. that might have to wait for mavenization. What's 0.94's penetration these days, does anyone know?
          Russell Jurney made changes -
          Description Updating the JAR in ivy/libraries.properties addresses the issue. Pig HBaseStorage doesn't work against a local mode HBase 0.94.1.

          Updating the JAR in ivy/libraries.properties addresses the issue, which is pasted here:

          2012-08-24 23:27:07,431 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2999: Unexpected internal error. Not a host:port pair: ?'58971@russell-jurneys-macbook-pro.local10.0.1.11,55171,1345873780529
          2012-08-24 23:27:07,431 [main] ERROR org.apache.pig.tools.grunt.Grunt - java.lang.IllegalArgumentException: Not a host:port pair: ?'58971@russell-jurneys-macbook-pro.local10.0.1.11,55171,1345873780529
          at org.apache.hadoop.hbase.HServerAddress.<init>(HServerAddress.java:60)
          at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.dataToHServerAddress(RootRegionTracker.java:82)
          at org.apache.hadoop.hbase.zookeeper.RootRegionTracker.waitRootRegionLocation(RootRegionTracker.java:73)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:575)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:684)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:586)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:684)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:590)
          at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:555)
          at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:171)
          at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:145)
          at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:196)
          at org.apache.pig.backend.hadoop.hbase.HBaseStorage.getOutputFormat(HBaseStorage.java:593)
          at org.apache.pig.newplan.logical.rules.InputOutputFileValidator$InputOutputFileVisitor.visit(InputOutputFileValidator.java:80)
          at org.apache.pig.newplan.logical.relational.LOStore.accept(LOStore.java:66)
          at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:64)
          at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
          at org.apache.pig.newplan.DepthFirstWalker.depthFirst(DepthFirstWalker.java:66)
          at org.apache.pig.newplan.DepthFirstWalker.walk(DepthFirstWalker.java:53)
          at org.apache.pig.newplan.PlanVisitor.visit(PlanVisitor.java:50)
          at org.apache.pig.newplan.logical.rules.InputOutputFileValidator.validate(InputOutputFileValidator.java:45)
          at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.compile(HExecutionEngine.java:288)
          at org.apache.pig.PigServer.compilePp(PigServer.java:1310)
          at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1247)
          at org.apache.pig.PigServer.execute(PigServer.java:1239)
          at org.apache.pig.PigServer.executeBatch(PigServer.java:333)
          at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:136)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:197)
          at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:169)
          at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
          at org.apache.pig.Main.run(Main.java:602)
          at org.apache.pig.Main.main(Main.java:154)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:597)
          at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
          Russell Jurney made changes -
          Field Original Value New Value
          Attachment PIG-2891.patch [ 12542401 ]
          Hide
          Russell Jurney added a comment -

          Updated version of HBase jar in ivy from 0.90.0 to 0.94.1

          Show
          Russell Jurney added a comment - Updated version of HBase jar in ivy from 0.90.0 to 0.94.1
          Russell Jurney created issue -

            People

            • Assignee:
              Alan Gates
              Reporter:
              Russell Jurney
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development