Pig
  1. Pig
  2. PIG-3285

Jobs using HBaseStorage fail to ship dependency jars

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
      None

      Description

      Launching a job consuming HBaseStorage fails out of the box. The user must specify -Dpig.additional.jars for HBase and all of its dependencies. Exceptions look something like this:

      2013-04-19 18:58:39,360 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoClassDefFoundError: com/google/protobuf/Message
      	at org.apache.hadoop.hbase.io.HbaseObjectWritable.<clinit>(HbaseObjectWritable.java:266)
      	at org.apache.hadoop.hbase.ipc.Invocation.write(Invocation.java:139)
      	at org.apache.hadoop.hbase.ipc.HBaseClient$Connection.sendParam(HBaseClient.java:612)
      	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:975)
      	at org.apache.hadoop.hbase.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:84)
      	at $Proxy7.getProtocolVersion(Unknown Source)
      	at org.apache.hadoop.hbase.ipc.WritableRpcEngine.getProxy(WritableRpcEngine.java:136)
      	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:208)
      
      1. 2.pig
        0.1 kB
        Nick Dimiduk
      2. 1.txt
        0.0 kB
        Nick Dimiduk
      3. 1.pig
        0.1 kB
        Nick Dimiduk
      4. 0001-PIG-3285-Add-HBase-dependency-jars.patch
        2 kB
        Nick Dimiduk
      5. 0001-PIG-3285-Add-HBase-dependency-jars.patch
        2 kB
        Nick Dimiduk
      6. 0001-PIG-3285-Add-HBase-dependency-jars.patch
        4 kB
        Nick Dimiduk
      7. 0001-PIG-3285-Add-HBase-dependency-jars.patch
        5 kB
        Nick Dimiduk
      8. 0001-PIG-3285-Add-HBase-dependency-jars.patch
        4 kB
        Nick Dimiduk
      9. 0001-PIG-3285-Add-HBase-dependency-jars.patch
        5 kB
        Nick Dimiduk

        Issue Links

          Activity

          Hide
          Nick Dimiduk added a comment -

          Attaching example scripts to demonstrate the issue. Repro steps look something like:

          1. hadoop -put 1.txt to your home directory.
          2. Make sure HBase is running.
          3. Execute 1.pig.
          Show
          Nick Dimiduk added a comment - Attaching example scripts to demonstrate the issue. Repro steps look something like: hadoop -put 1.txt to your home directory. Make sure HBase is running. Execute 1.pig.
          Hide
          Daniel Dai added a comment -

          The problem is we don't ship protobuf.jar, here is the code:

          TableMapReduceUtil.addDependencyJars(job.getConfiguration(),
                      org.apache.hadoop.hbase.client.HTable.class,
                      com.google.common.collect.Lists.class,
                      org.apache.zookeeper.ZooKeeper.class)
          

          One other issue is we don't have protobuf.jar in Pig codebase. If we add a protobuf class, Pig cannot compile. We would need to add a compilation dependency for protobuf.

          Show
          Daniel Dai added a comment - The problem is we don't ship protobuf.jar, here is the code: TableMapReduceUtil.addDependencyJars(job.getConfiguration(), org.apache.hadoop.hbase.client.HTable.class, com.google.common.collect.Lists.class, org.apache.zookeeper.ZooKeeper.class) One other issue is we don't have protobuf.jar in Pig codebase. If we add a protobuf class, Pig cannot compile. We would need to add a compilation dependency for protobuf.
          Hide
          Nick Dimiduk added a comment -

          This patch invokes the correct incantation of TableMapReduceUtils#addDependencyJars. Specifically, it allows HBase to add the jars to the job on Pig's behalf. Note that the HBase jars must be on the Pig classpath for job launching to succeed. This is provided via PIG-2786 and can be invoked by defining environment variables for HBASE_HOME and HBASE_CONF_DIR.

          Show
          Nick Dimiduk added a comment - This patch invokes the correct incantation of TableMapReduceUtils#addDependencyJars . Specifically, it allows HBase to add the jars to the job on Pig's behalf. Note that the HBase jars must be on the Pig classpath for job launching to succeed. This is provided via PIG-2786 and can be invoked by defining environment variables for HBASE_HOME and HBASE_CONF_DIR .
          Hide
          Nick Dimiduk added a comment -

          The reason this patch works is that protobuf et al are shipped with HBase and are available on the launch script's classpath via $HBASE_HOME.

          Show
          Nick Dimiduk added a comment - The reason this patch works is that protobuf et al are shipped with HBase and are available on the launch script's classpath via $HBASE_HOME .
          Hide
          Daniel Dai added a comment -

          TableMapReduce.addDependencyJars(job) works for HBase 0.95+. For HBase 0.94, there is a bug which does not add hbase.jar. Also a side effect for addDependencyJars(job) is it adds hadoop.jar and pig.jar into tmpjars. Both of which are already taken care of by Pig. I am not sure if we double ship those jars if we doing this. Actually I would prefer a TableMapReduce.addDependencyJars version which only adds hbase.jar/guava.jar/protobuf.jar and additional dependencies when hbase evolves (but no hadoop.jar/pig.jar)

          Show
          Daniel Dai added a comment - TableMapReduce.addDependencyJars(job) works for HBase 0.95+. For HBase 0.94, there is a bug which does not add hbase.jar. Also a side effect for addDependencyJars(job) is it adds hadoop.jar and pig.jar into tmpjars. Both of which are already taken care of by Pig. I am not sure if we double ship those jars if we doing this. Actually I would prefer a TableMapReduce.addDependencyJars version which only adds hbase.jar/guava.jar/protobuf.jar and additional dependencies when hbase evolves (but no hadoop.jar/pig.jar)
          Hide
          Nick Dimiduk added a comment -

          Ah, right. This bug was fixed in HBASE-8146, which will go out with hbase-0.94.7. I'll update the patch to smooth over both versions.

          Show
          Nick Dimiduk added a comment - Ah, right. This bug was fixed in HBASE-8146 , which will go out with hbase-0.94.7. I'll update the patch to smooth over both versions.
          Hide
          Nick Dimiduk added a comment -

          Updated patch smoothes over hbase-0.94 versions.

          Show
          Nick Dimiduk added a comment - Updated patch smoothes over hbase-0.94 versions.
          Hide
          Nick Dimiduk added a comment -

          bump.

          Show
          Nick Dimiduk added a comment - bump.
          Hide
          Rohini Palaniswamy added a comment -

          I am not sure if we double ship those jars if we doing this. Actually I would prefer a TableMapReduce.addDependencyJars version which only adds hbase.jar/guava.jar/protobuf.jar and additional dependencies when hbase evolves (but no hadoop.jar/pig.jar)

          Nick, Looking at the code I am sure we will end up double shipping jars which is very inefficient. It would be good to write a separate function instead of TableMapReduce.addDependencyJars(job) that filters out pig and hadoop jars (classes starting with org.apache.pig and org.apache.hadoop) and those in pigContext.extraJars from the list of classes in TableMapReduce.addDependencyJars(job) and then set them on tmpjars. You can reuse JarManager.findContainingJar to find the jar for a class file.

          Show
          Rohini Palaniswamy added a comment - I am not sure if we double ship those jars if we doing this. Actually I would prefer a TableMapReduce.addDependencyJars version which only adds hbase.jar/guava.jar/protobuf.jar and additional dependencies when hbase evolves (but no hadoop.jar/pig.jar) Nick, Looking at the code I am sure we will end up double shipping jars which is very inefficient. It would be good to write a separate function instead of TableMapReduce.addDependencyJars(job) that filters out pig and hadoop jars (classes starting with org.apache.pig and org.apache.hadoop) and those in pigContext.extraJars from the list of classes in TableMapReduce.addDependencyJars(job) and then set them on tmpjars. You can reuse JarManager.findContainingJar to find the jar for a class file.
          Hide
          Nick Dimiduk added a comment -

          Good point, Rohini Palaniswamy. Would it be instead relevant for Hadoop to remove duplicates in the "tmpjars" list?

          Also tangentially relevant, have a look at HBASE-8438 for building a minimal set of classpath additions. This will minimize the number of places jars can be pulled from.

          Show
          Nick Dimiduk added a comment - Good point, Rohini Palaniswamy . Would it be instead relevant for Hadoop to remove duplicates in the "tmpjars" list? Also tangentially relevant, have a look at HBASE-8438 for building a minimal set of classpath additions. This will minimize the number of places jars can be pulled from.
          Hide
          Rohini Palaniswamy added a comment -

          Nick,
          You can raise a jira in Hadoop to handle duplicates. But since we support all older versions, we can't rely on it. Also it will not help with the current problem anyways.

          The problem here is that hbase code is setting some jars in tmpjars which copies the jar to hdfs to /user/[username]/.staging and adds that hdfs file to DistributedCache.addArchiveToClassPath when JobClient.submitJob() is done. Pig already puts the pig.jar as job.jar and it ships the other registered jar to a tmp location in hdfs (/tmp/...) and then does a DistributedCache.addFileToClassPath before submitting the job. In this case, all the three settings are different and since pig does not use tmpfiles or tmpjars and does the work by itself the hdfs path is also different. So duplicates have to be resolved at the pig level.

          Show
          Rohini Palaniswamy added a comment - Nick, You can raise a jira in Hadoop to handle duplicates. But since we support all older versions, we can't rely on it. Also it will not help with the current problem anyways. The problem here is that hbase code is setting some jars in tmpjars which copies the jar to hdfs to /user/ [username] /.staging and adds that hdfs file to DistributedCache.addArchiveToClassPath when JobClient.submitJob() is done. Pig already puts the pig.jar as job.jar and it ships the other registered jar to a tmp location in hdfs (/tmp/...) and then does a DistributedCache.addFileToClassPath before submitting the job. In this case, all the three settings are different and since pig does not use tmpfiles or tmpjars and does the work by itself the hdfs path is also different. So duplicates have to be resolved at the pig level.
          Hide
          Nick Dimiduk added a comment -

          Gross. Too many code-paths. How then to proceed? Can we consolidate on a single, approved method of shipping job dependencies?

          (cc Andrew Purtell since you were interested in the related HIVE-2055)

          Show
          Nick Dimiduk added a comment - Gross. Too many code-paths. How then to proceed? Can we consolidate on a single, approved method of shipping job dependencies? (cc Andrew Purtell since you were interested in the related HIVE-2055 )
          Hide
          Rohini Palaniswamy added a comment -

          Not too many code paths. For hbase storage, need to write a method that

          • Filters out pig and hadoop classes from the list of classes so that pig and hadoop jar are not included.
          • Find the jars for the other classes and filter out any jars already present in PigContext.extrajars and add only the rest to tmpjars.

          Also ensure that you add HTable.class apart from Zookeeper, inputformat, input/output key/value, partitioner and combiner classes.

          Show
          Rohini Palaniswamy added a comment - Not too many code paths. For hbase storage, need to write a method that Filters out pig and hadoop classes from the list of classes so that pig and hadoop jar are not included. Find the jars for the other classes and filter out any jars already present in PigContext.extrajars and add only the rest to tmpjars. Also ensure that you add HTable.class apart from Zookeeper, inputformat, input/output key/value, partitioner and combiner classes.
          Hide
          Nick Dimiduk added a comment -

          Not too many code paths.

          Sure there are. Both Pig and HBase are replicating the behavior of ToolRunner's libjars argument for including jars with a job. They do so in slightly different ways, but thus we have 3 different code-paths. I'd prefer consolidation on a single code-path.

          Filters out pig and hadoop classes from the list of classes so that pig and hadoop jar are not included.

          We can add a method, something like addHBaseDependencyJars(Job) which will add only HBase and it's dependency jars (currently: zookeeper, protobuf, guava), nothing else. That way, we're not including any redundant Pig or Hadoop jars and HBase is managing it's own dependencies (meaning Pig won't have to change every time we change something). This is effectively the same doing what you say above, "Also ensure that you add HTable.class apart from Zookeeper, inputformat, input/output key/value, partitioner and combiner classes," that is, omitting inputformat, keys, values, partitioner, combiner. Does that sound like it'll accomplish what this filter intends?

          Find the jars for the other classes and filter out any jars already present in PigContext.extrajars and add only the rest to tmpjars.

          How do we access the PigContext? Is it in the jobConf or some such? I'd rather not put Pig-specific code in the bowels of HBase mapreduce code; my preference is to build generic APIs that can be used across the board.

          HBase APIs are designed to assist people writing raw MR jobs against HBase (ie, including key/value classes, input/output format classes, &c). The slightly different requirements of Pig and Hive need to be addressed as well.

          Show
          Nick Dimiduk added a comment - Not too many code paths. Sure there are. Both Pig and HBase are replicating the behavior of ToolRunner 's libjars argument for including jars with a job. They do so in slightly different ways, but thus we have 3 different code-paths. I'd prefer consolidation on a single code-path. Filters out pig and hadoop classes from the list of classes so that pig and hadoop jar are not included. We can add a method, something like addHBaseDependencyJars(Job) which will add only HBase and it's dependency jars (currently: zookeeper, protobuf, guava), nothing else. That way, we're not including any redundant Pig or Hadoop jars and HBase is managing it's own dependencies (meaning Pig won't have to change every time we change something). This is effectively the same doing what you say above, "Also ensure that you add HTable.class apart from Zookeeper, inputformat, input/output key/value, partitioner and combiner classes," that is, omitting inputformat, keys, values, partitioner, combiner. Does that sound like it'll accomplish what this filter intends? Find the jars for the other classes and filter out any jars already present in PigContext.extrajars and add only the rest to tmpjars. How do we access the PigContext? Is it in the jobConf or some such? I'd rather not put Pig-specific code in the bowels of HBase mapreduce code; my preference is to build generic APIs that can be used across the board. HBase APIs are designed to assist people writing raw MR jobs against HBase (ie, including key/value classes, input/output format classes, &c). The slightly different requirements of Pig and Hive need to be addressed as well.
          Hide
          Rohini Palaniswamy added a comment -

          Actually I got a little confused with the patch as TableMapReduce.addDependencyJars(job) was adding all those and focused just on Daniel's comment. If your intention is to only add protobuf jar, you can do a Class.forName(some protobuf class name) and if that does not throw a CNFE (meaning older hbase versions) you can pass that class also to the TableMapreduceUtil.addDependencyJars(Configuration conf, Class... classes)

          Show
          Rohini Palaniswamy added a comment - Actually I got a little confused with the patch as TableMapReduce.addDependencyJars(job) was adding all those and focused just on Daniel's comment. If your intention is to only add protobuf jar, you can do a Class.forName(some protobuf class name) and if that does not throw a CNFE (meaning older hbase versions) you can pass that class also to the TableMapreduceUtil.addDependencyJars(Configuration conf, Class... classes)
          Hide
          Daniel Dai added a comment -

          Agree with Rohini, that should be a simple fix to add protobuf.jar. We don't need to double ship jars and make things complicated.

          Show
          Daniel Dai added a comment - Agree with Rohini, that should be a simple fix to add protobuf.jar. We don't need to double ship jars and make things complicated.
          Hide
          Nick Dimiduk added a comment -

          I don't want this to become a game of dependency tracking across projects. That said, HBase doesn't add dependencies all that often, so maybe it doesn't matter in practice.

          Show
          Nick Dimiduk added a comment - I don't want this to become a game of dependency tracking across projects. That said, HBase doesn't add dependencies all that often, so maybe it doesn't matter in practice.
          Hide
          Daniel Dai added a comment -

          If HBase adding another method only adding hbase.jar/guava.jar/protobuf.jar etc(but not inputformat/outputformat jar), Pig can switch to use that. But for now, TableMapReduce.addDependencyJars(job) is not what we want.

          Show
          Daniel Dai added a comment - If HBase adding another method only adding hbase.jar/guava.jar/protobuf.jar etc(but not inputformat/outputformat jar), Pig can switch to use that. But for now, TableMapReduce.addDependencyJars(job) is not what we want.
          Hide
          Rohini Palaniswamy added a comment -

          Canceling patch for now so that it does not show in Patch Available list.

          Show
          Rohini Palaniswamy added a comment - Canceling patch for now so that it does not show in Patch Available list.
          Hide
          Nick Dimiduk added a comment -

          Hi Daniel Dai. Have a look at the patch on HBASE-9165. It include a new method for this use-case: TableMapReduceUtil#addHBaseDependencyJars(Configuration).

          Show
          Nick Dimiduk added a comment - Hi Daniel Dai . Have a look at the patch on HBASE-9165 . It include a new method for this use-case: TableMapReduceUtil#addHBaseDependencyJars(Configuration).
          Hide
          Daniel Dai added a comment -

          That sounds good. We can switch to it for newer version of hbase once HBASE-9165 committed.

          Show
          Daniel Dai added a comment - That sounds good. We can switch to it for newer version of hbase once HBASE-9165 committed.
          Hide
          Nick Dimiduk added a comment -

          Daniel Dai would you mind commenting positively on the HBase ticket as well? Thanks.

          Show
          Nick Dimiduk added a comment - Daniel Dai would you mind commenting positively on the HBase ticket as well? Thanks.
          Hide
          Nick Dimiduk added a comment -

          FYI HBASE-9165 is committed to 0.96 and trunk. It will be available with HBase 0.96.1. I hope to also backport it to 0.94, available on 0.94.14 and later. The new method is TableMapReduceUtil#addHBaseDependencyJars(Configuration). I'm now using it in a new hbase bin script command mapredcp, as of HBASE-8438. Its output is as follows:

          $ ./bin/hbase mapredcp | tr ':' '\n'
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/netty-3.6.6.Final.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-hadoop-compat-0.97.0-SNAPSHOT.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/protobuf-java-2.5.0.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/guava-12.0.1.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/htrace-core-2.01.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-protocol-0.97.0-SNAPSHOT.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-client-0.97.0-SNAPSHOT.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/zookeeper-3.4.5.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-server-0.97.0-SNAPSHOT.jar
          /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-common-0.97.0-SNAPSHOT.jar
          
          Show
          Nick Dimiduk added a comment - FYI HBASE-9165 is committed to 0.96 and trunk. It will be available with HBase 0.96.1. I hope to also backport it to 0.94, available on 0.94.14 and later. The new method is TableMapReduceUtil#addHBaseDependencyJars(Configuration) . I'm now using it in a new hbase bin script command mapredcp , as of HBASE-8438 . Its output is as follows: $ ./bin/hbase mapredcp | tr ':' '\n' /private/tmp/hbase-0.97.0-SNAPSHOT/lib/netty-3.6.6.Final.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-hadoop-compat-0.97.0-SNAPSHOT.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/protobuf-java-2.5.0.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/guava-12.0.1.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/htrace-core-2.01.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-protocol-0.97.0-SNAPSHOT.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-client-0.97.0-SNAPSHOT.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/zookeeper-3.4.5.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-server-0.97.0-SNAPSHOT.jar /private/tmp/hbase-0.97.0-SNAPSHOT/lib/hbase-common-0.97.0-SNAPSHOT.jar
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #838 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/838/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341)
          • /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #838 (See https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/838/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341) /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Hide
          Nick Dimiduk added a comment -

          Backport to HBase 0.94.14 is complete.

          Show
          Nick Dimiduk added a comment - Backport to HBase 0.94.14 is complete.
          Hide
          Nick Dimiduk added a comment -

          Here's an updated patch. It would be nice to test and confirm end-to-end. Testing in my local checkout results in an NPE; I assume I'm doing something wrong.

          $ ant clean && ant && ant -Dtestcase=TestHBaseStorage test
          ...
          Testcase: org.apache.pig.test.TestHBaseStorage took 0 sec
                  Caused an ERROR
          null not an instance of org.apache.hadoop.hbase.MiniHBaseCluster
          java.lang.RuntimeException: null not an instance of org.apache.hadoop.hbase.MiniHBaseCluster
                  at org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:701)
                  at org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:1591)
                  at org.apache.pig.test.TestHBaseStorage.oneTimeTearDown(TestHBaseStorage.java:92)
          
          Show
          Nick Dimiduk added a comment - Here's an updated patch. It would be nice to test and confirm end-to-end. Testing in my local checkout results in an NPE; I assume I'm doing something wrong. $ ant clean && ant && ant -Dtestcase=TestHBaseStorage test ... Testcase: org.apache.pig.test.TestHBaseStorage took 0 sec Caused an ERROR null not an instance of org.apache.hadoop.hbase.MiniHBaseCluster java.lang.RuntimeException: null not an instance of org.apache.hadoop.hbase.MiniHBaseCluster at org.apache.hadoop.hbase.HBaseTestingUtility.getMiniHBaseCluster(HBaseTestingUtility.java:701) at org.apache.hadoop.hbase.HBaseTestingUtility.getHBaseCluster(HBaseTestingUtility.java:1591) at org.apache.pig.test.TestHBaseStorage.oneTimeTearDown(TestHBaseStorage.java:92)
          Hide
          Jarek Jarcec Cecho added a comment -

          I really do like idea behind this patch, thank you for driving the effort on both sides Nick Dimiduk! Do you think that we can leave the older code laying around for couple of Pig releases and switch between those two conditionally based on the HBase version? I'm concerned a bit about backward compatibility here as this patch will make pig not working on a HBase releases where it is currently working.

          Show
          Jarek Jarcec Cecho added a comment - I really do like idea behind this patch, thank you for driving the effort on both sides Nick Dimiduk ! Do you think that we can leave the older code laying around for couple of Pig releases and switch between those two conditionally based on the HBase version? I'm concerned a bit about backward compatibility here as this patch will make pig not working on a HBase releases where it is currently working.
          Hide
          Nick Dimiduk added a comment -

          More compatibility is better. I think that can be done with a little reflection magic. That logic will be pretty hairy as it spans at least 4 HBase releases. I'm not familiar with Pig internals so I'll leave a more sophisticated patch up to a maintainer (unless someone can illustrate how I can test my patches).

          Show
          Nick Dimiduk added a comment - More compatibility is better. I think that can be done with a little reflection magic. That logic will be pretty hairy as it spans at least 4 HBase releases. I'm not familiar with Pig internals so I'll leave a more sophisticated patch up to a maintainer (unless someone can illustrate how I can test my patches).
          Hide
          Jarek Jarcec Cecho added a comment -

          Yeah I agree, more compatibility is definitely better. Considering that the class TableMapReduceUtil have "always" been there, it might become fairly straightforward. Do you think that following snippet might work?

          Method m = TableMapReduceUtil.class.getMethod("addHBaseDependencyJars", Configuration.class);
          if(m != null) {
            m.invoke(null, job.getConfiguration());
          } else {
            // Old code
          }
          

          And then the basic sanity testing would be with changing the HBase version on the command line, for example:

          ant clean test -Dtestcase=TestHBaseStorage -Dhbase.version=0.94.1
          ant clean test -Dtestcase=TestHBaseStorage -Dhbase.version=0.94.14
          
          Show
          Jarek Jarcec Cecho added a comment - Yeah I agree, more compatibility is definitely better. Considering that the class TableMapReduceUtil have "always" been there, it might become fairly straightforward. Do you think that following snippet might work? Method m = TableMapReduceUtil.class.getMethod( "addHBaseDependencyJars" , Configuration.class); if (m != null ) { m.invoke( null , job.getConfiguration()); } else { // Old code } And then the basic sanity testing would be with changing the HBase version on the command line, for example: ant clean test -Dtestcase=TestHBaseStorage -Dhbase.version=0.94.1 ant clean test -Dtestcase=TestHBaseStorage -Dhbase.version=0.94.14
          Hide
          Nick Dimiduk added a comment -

          I suspect that would work, but I've been bitten enough by the nuances of indirect dependency resolution at runtime that I'll withhold judgement until running a job on a real cluster

          You'll also need to add permutations for 0.96.0 and 0.96.1. At least with this single API, we should be able to iterate on our side without repercussions for Pig.

          Show
          Nick Dimiduk added a comment - I suspect that would work, but I've been bitten enough by the nuances of indirect dependency resolution at runtime that I'll withhold judgement until running a job on a real cluster You'll also need to add permutations for 0.96.0 and 0.96.1. At least with this single API, we should be able to iterate on our side without repercussions for Pig.
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-TRUNK #4681 (See https://builds.apache.org/job/HBase-TRUNK/4681/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341)
          • /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          • /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-TRUNK #4681 (See https://builds.apache.org/job/HBase-TRUNK/4681/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. (ndimiduk: rev 1542341) /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceUtil.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96 #190 (See https://builds.apache.org/job/hbase-0.96/190/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce.

          Note this patch differs from the one applied to TRUNK/0.98 in that it omits
          changes to TestTableMapReduceUtil which have not yet been backported from
          HBASE-8534. This can be addressed when that patch is backported (HBASE-9484). (ndimiduk: rev 1542342)

          • /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96 #190 (See https://builds.apache.org/job/hbase-0.96/190/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. Note this patch differs from the one applied to TRUNK/0.98 in that it omits changes to TestTableMapReduceUtil which have not yet been backported from HBASE-8534 . This can be addressed when that patch is backported ( HBASE-9484 ). (ndimiduk: rev 1542342) /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in HBase-0.94 #1203 (See https://builds.apache.org/job/HBase-0.94/1203/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          Separate adding HBase and dependencies from adding other job dependencies, and
          expose it as a separate method that other projects can use (for PIG-3285,
          HIVE-2055). (ndimiduk: rev 1542414)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - FAILURE: Integrated in HBase-0.94 #1203 (See https://builds.apache.org/job/HBase-0.94/1203/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 , HIVE-2055 ). (ndimiduk: rev 1542414) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in HBase-0.94-security #337 (See https://builds.apache.org/job/HBase-0.94-security/337/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          Separate adding HBase and dependencies from adding other job dependencies, and
          expose it as a separate method that other projects can use (for PIG-3285,
          HIVE-2055). (ndimiduk: rev 1542414)

          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Show
          Hudson added a comment - SUCCESS: Integrated in HBase-0.94-security #337 (See https://builds.apache.org/job/HBase-0.94-security/337/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 , HIVE-2055 ). (ndimiduk: rev 1542414) /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in hbase-0.96-hadoop2 #120 (See https://builds.apache.org/job/hbase-0.96-hadoop2/120/)
          HBASE-9165 [mapreduce] Modularize building dependency jars

          • Separate adding HBase and dependencies from adding other job
            dependencies, and expose it as a separate method that other
            projects can use (for PIG-3285).
          • Explicitly add hbase-server to the list of dependencies we ship
            with the job, for users who extend the classes we provide (see
            HBASE-9112).
          • Add integration test for addDependencyJars.
          • Code reuse for TestTableMapReduce.

          Note this patch differs from the one applied to TRUNK/0.98 in that it omits
          changes to TestTableMapReduceUtil which have not yet been backported from
          HBASE-8534. This can be addressed when that patch is backported (HBASE-9484). (ndimiduk: rev 1542342)

          • /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java
          • /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
          • /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Show
          Hudson added a comment - FAILURE: Integrated in hbase-0.96-hadoop2 #120 (See https://builds.apache.org/job/hbase-0.96-hadoop2/120/ ) HBASE-9165 [mapreduce] Modularize building dependency jars Separate adding HBase and dependencies from adding other job dependencies, and expose it as a separate method that other projects can use (for PIG-3285 ). Explicitly add hbase-server to the list of dependencies we ship with the job, for users who extend the classes we provide (see HBASE-9112 ). Add integration test for addDependencyJars. Code reuse for TestTableMapReduce. Note this patch differs from the one applied to TRUNK/0.98 in that it omits changes to TestTableMapReduceUtil which have not yet been backported from HBASE-8534 . This can be addressed when that patch is backported ( HBASE-9484 ). (ndimiduk: rev 1542342) /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/mapreduce/IntegrationTestTableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/IdentityTableMapper.java /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
          Hide
          Jarek Jarcec Cecho added a comment -

          Yeah, I hear your concern Nick Dimiduk. Nevertheless I'll be more than happy to work with you to ensure that all the required combinations will work. I have to say though that I'm not a Pig committer and thus we will need someone else to do the commit.

          Show
          Jarek Jarcec Cecho added a comment - Yeah, I hear your concern Nick Dimiduk . Nevertheless I'll be more than happy to work with you to ensure that all the required combinations will work. I have to say though that I'm not a Pig committer and thus we will need someone else to do the commit.
          Hide
          Cheolsoo Park added a comment -

          Jarek Jarcec Cecho, please upload a new patch that includes what you suggest. Committing it is not a problem at all.

          Show
          Cheolsoo Park added a comment - Jarek Jarcec Cecho , please upload a new patch that includes what you suggest. Committing it is not a problem at all.
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you for your support Cheolsoo Park!

          Do you feel comfortable with making the suggested changes to the patch Nick Dimiduk or do you want me to jump in and do them?

          Show
          Jarek Jarcec Cecho added a comment - Thank you for your support Cheolsoo Park ! Do you feel comfortable with making the suggested changes to the patch Nick Dimiduk or do you want me to jump in and do them?
          Hide
          Nick Dimiduk added a comment -

          Here's an updated patch that does as you suggest.

          Show
          Nick Dimiduk added a comment - Here's an updated patch that does as you suggest.
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Nick Dimiduk! Couple of high level notes:

          -      TableMapReduceUtil.addDependencyJars(job.getConfiguration(), klass);
          

          It seems that we've accidentally removed the call TableMapReduceUtil.addDependencyJars() from method addClassToJobIfExists(). As a result not all required jars are propagated to the DistributedCache when running on HBase 0.95+.

          +            // method doesn't exist or invocation failed. moving on.
          

          Can we at least do a debug log message stating that TableMapReduceUtil.addHBaseDependencyJars is not available and thus we will revert to our own logic for propagating jars? It think that it will help to know what logic we used for propagating jars while investigating failing pig scripts.

          Show
          Jarek Jarcec Cecho added a comment - Thank you Nick Dimiduk ! Couple of high level notes: - TableMapReduceUtil.addDependencyJars(job.getConfiguration(), klass); It seems that we've accidentally removed the call TableMapReduceUtil.addDependencyJars() from method addClassToJobIfExists() . As a result not all required jars are propagated to the DistributedCache when running on HBase 0.95+. + // method doesn't exist or invocation failed. moving on. Can we at least do a debug log message stating that TableMapReduceUtil.addHBaseDependencyJars is not available and thus we will revert to our own logic for propagating jars? It think that it will help to know what logic we used for propagating jars while investigating failing pig scripts.
          Hide
          Nick Dimiduk added a comment -

          Thanks for the review, Jarek Jarcec Cecho. Good catch on the unintended change to addClassToJobIfExists.

          Attaching an updated patch addressing both comments.

          Show
          Nick Dimiduk added a comment - Thanks for the review, Jarek Jarcec Cecho . Good catch on the unintended change to addClassToJobIfExists. Attaching an updated patch addressing both comments.
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you for the quick turnaround Nick Dimiduk!

          I was able to run TestHBaseStorage unit tests for all the combinations using following commands:

          # Old path:
          ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage
          ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage  -Dprotobuf-java.version=2.5.0
          
          # New path:
          
          # Compile HBase 0.94 branch and install it to local maven cache (mvn clean install -DskipTests)
          ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage -Dhbase94.version=0.94.14
          # Compile HBase 0.96 branch and install it to local maven cache (mvn clean install -DskipTests)
          ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage  -Dprotobuf-java.version=2.5.0 -Dhbase95.version=0.96.1-SNAPSHOT
          

          The only problem that I've encounter so far is the exception ReflectiveOperationException (javadoc) that has been added in JDK7, while Pig still requires JDK6 compatibility. I've substituted the exception with Exception for the purpose of above tests.

          I'll try to give it a spin on a real clusters to see if everything will work as expected.

          Show
          Jarek Jarcec Cecho added a comment - Thank you for the quick turnaround Nick Dimiduk ! I was able to run TestHBaseStorage unit tests for all the combinations using following commands: # Old path: ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage -Dprotobuf-java.version=2.5.0 # New path: # Compile HBase 0.94 branch and install it to local maven cache (mvn clean install -DskipTests) ant clean test -Dhbaseversion=94 -Dtestcase=TestHBaseStorage -Dhbase94.version=0.94.14 # Compile HBase 0.96 branch and install it to local maven cache (mvn clean install -DskipTests) ant clean test -Dhbaseversion=95 -Dtestcase=TestHBaseStorage -Dprotobuf-java.version=2.5.0 -Dhbase95.version=0.96.1-SNAPSHOT The only problem that I've encounter so far is the exception ReflectiveOperationException ( javadoc ) that has been added in JDK7, while Pig still requires JDK6 compatibility. I've substituted the exception with Exception for the purpose of above tests. I'll try to give it a spin on a real clusters to see if everything will work as expected.
          Hide
          Nick Dimiduk added a comment -

          Excellent. I didn't realize ReflectiveOperationException was introduced in JDK7, nice catch. Thanks for taking this the last mile!

          Show
          Nick Dimiduk added a comment - Excellent. I didn't realize ReflectiveOperationException was introduced in JDK7, nice catch. Thanks for taking this the last mile!
          Hide
          Jarek Jarcec Cecho added a comment -

          I've tested the change on a real clusters running HBase 0.94 with and without the HBASE-9165 and everything seems to be working just fine. It would be great to also test 0.96 with and without the change, but I do not feel that it's entirely necessary. I'm +1 on the patch, provided that we will remove the JDK7 only class ReflectiveOperationException.

          Show
          Jarek Jarcec Cecho added a comment - I've tested the change on a real clusters running HBase 0.94 with and without the HBASE-9165 and everything seems to be working just fine. It would be great to also test 0.96 with and without the change, but I do not feel that it's entirely necessary. I'm +1 on the patch, provided that we will remove the JDK7 only class ReflectiveOperationException .
          Hide
          Nick Dimiduk added a comment -

          Updating patch to not use ReflectiveOperationException.

          Thanks for testing! Would you mind taking it for a spin with hbase-0.94.14RC1? It was just posted and includes all necessary patches required to test the new logic.

          http://people.apache.org/~larsh/hbase-0.94.14-rc1/

          Show
          Nick Dimiduk added a comment - Updating patch to not use ReflectiveOperationException. Thanks for testing! Would you mind taking it for a spin with hbase-0.94.14RC1? It was just posted and includes all necessary patches required to test the new logic. http://people.apache.org/~larsh/hbase-0.94.14-rc1/
          Hide
          Jarek Jarcec Cecho added a comment -

          My apologies for the delay Nick Dimiduk. I've tried the latest patch against linked HBase RC and everything seems to be working.

          +1 (non-binding)

          Feel free to take a look Cheolsoo Park!

          Show
          Jarek Jarcec Cecho added a comment - My apologies for the delay Nick Dimiduk . I've tried the latest patch against linked HBase RC and everything seems to be working. +1 (non-binding) Feel free to take a look Cheolsoo Park !
          Hide
          Cheolsoo Park added a comment -

          +1. I ran TestHBaseStorage myself, and I trust Jarcec's test.

          Show
          Cheolsoo Park added a comment - +1. I ran TestHBaseStorage myself, and I trust Jarcec's test.
          Hide
          Cheolsoo Park added a comment -

          Committed to trunk. Thank you Nick and Jarcec!

          Show
          Cheolsoo Park added a comment - Committed to trunk. Thank you Nick and Jarcec!
          Hide
          Jarek Jarcec Cecho added a comment -

          Thank you Cheolsoo Park!

          Show
          Jarek Jarcec Cecho added a comment - Thank you Cheolsoo Park !

            People

            • Assignee:
              Nick Dimiduk
              Reporter:
              Nick Dimiduk
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development