Hive
  1. Hive
  2. HIVE-2468

Make Hive compile against Hadoop 0.23

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      Due to restructure of Hadoop 0.22 branch compared to Hadoop 0.20 Hive does not compile against 0.22

      1. HIVE-2468.patch
        77 kB
        Tom White
      2. HIVE-2468.patch
        78 kB
        Tom White
      3. HIVE-2468.patch
        74 kB
        Tom White
      4. HIVE-2468.patch
        49 kB
        Tom White
      5. HIVE-2468.patch
        60 kB
        Tom White

        Issue Links

          Activity

          Hide
          Tom White added a comment -

          Here's a patch that Roman Shaposhnik and I wrote to get Hive compiling against Hadoop 0.23 (I haven't tried it against 0.22).

          Show
          Tom White added a comment - Here's a patch that Roman Shaposhnik and I wrote to get Hive compiling against Hadoop 0.23 (I haven't tried it against 0.22).
          Hide
          Ashutosh Chauhan added a comment -

          @Tom,
          This is very useful. Can you please create RB for it? Also, would you like to change the jira title to reflect it is for 0.23

          Show
          Ashutosh Chauhan added a comment - @Tom, This is very useful. Can you please create RB for it? Also, would you like to change the jira title to reflect it is for 0.23
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2762/
          -----------------------------------------------------------

          Review request for hive.

          Summary
          -------

          See https://issues.apache.org/jira/browse/HIVE-2468

          This addresses bug HIVE-2468.
          https://issues.apache.org/jira/browse/HIVE-2468

          Diffs


          trunk/build.properties 1196775
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775
          trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775
          trunk/shims/build.xml 1196775
          trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775
          trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775
          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION
          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775
          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775

          Diff: https://reviews.apache.org/r/2762/diff

          Testing
          -------

          Thanks,

          Tom

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ ----------------------------------------------------------- Review request for hive. Summary ------- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468 . https://issues.apache.org/jira/browse/HIVE-2468 Diffs trunk/build.properties 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775 trunk/shims/build.xml 1196775 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775 trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775 trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775 Diff: https://reviews.apache.org/r/2762/diff Testing ------- Thanks, Tom
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2762/#review3316
          -----------------------------------------------------------

          Looks good overall. Please update build.properties and resubmit. Thanks!

          trunk/build.properties
          <https://reviews.apache.org/r/2762/#comment7404>

          For the time being we need to stick with 0.20.1 as the default Hadoop version against which Hive is tested, but we should start using 0.23.0 for the security tests, and should ensure that all tests pass when run against 0.23.0.

          Please set hadoop.version to 0.20.1 and hadoop.security.version to 0.23.0, and ensure that all tests pass when run as "ant test -Dhadoop.version=0.23.0".

          Also, I added 0.23.0 to the cloudera and facebook hive-deps mirrors, so please reference those locations instead.

          • Carl

          On 2011-11-07 23:55:47, Tom White wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2762/

          -----------------------------------------------------------

          (Updated 2011-11-07 23:55:47)

          Review request for hive.

          Summary

          -------

          See https://issues.apache.org/jira/browse/HIVE-2468

          This addresses bug HIVE-2468.

          https://issues.apache.org/jira/browse/HIVE-2468

          Diffs

          -----

          trunk/build.properties 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775

          trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775

          trunk/shims/build.xml 1196775

          trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775

          trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775

          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION

          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775

          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775

          Diff: https://reviews.apache.org/r/2762/diff

          Testing

          -------

          Thanks,

          Tom

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/#review3316 ----------------------------------------------------------- Looks good overall. Please update build.properties and resubmit. Thanks! trunk/build.properties < https://reviews.apache.org/r/2762/#comment7404 > For the time being we need to stick with 0.20.1 as the default Hadoop version against which Hive is tested, but we should start using 0.23.0 for the security tests, and should ensure that all tests pass when run against 0.23.0. Please set hadoop.version to 0.20.1 and hadoop.security.version to 0.23.0, and ensure that all tests pass when run as "ant test -Dhadoop.version=0.23.0". Also, I added 0.23.0 to the cloudera and facebook hive-deps mirrors, so please reference those locations instead. Carl On 2011-11-07 23:55:47, Tom White wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ ----------------------------------------------------------- (Updated 2011-11-07 23:55:47) Review request for hive. Summary ------- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468 . https://issues.apache.org/jira/browse/HIVE-2468 Diffs ----- trunk/build.properties 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775 trunk/shims/build.xml 1196775 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775 trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775 trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775 Diff: https://reviews.apache.org/r/2762/diff Testing ------- Thanks, Tom
          Hide
          Amareshwari Sriramadasu added a comment -

          Would need following changes in bin/hive:
          -if [ $hadoop_minor_ver -ne 20 -o $hadoop_patch_ver -eq 0 ]; then
          +if [ "$hadoop_minor_ver$hadoop_patch_ver" -lt "201" ]; then

          Also, I had to do changes in CombineFileInputFormatShim to work with 0.23. Because the following casting does not work anymore.

                CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, numSplits);
          

          And the paths of hadoop jars build.properties should be changed according to the 0.23 tar ball

          Show
          Amareshwari Sriramadasu added a comment - Would need following changes in bin/hive: -if [ $hadoop_minor_ver -ne 20 -o $hadoop_patch_ver -eq 0 ]; then +if [ "$hadoop_minor_ver$hadoop_patch_ver" -lt "201" ]; then Also, I had to do changes in CombineFileInputFormatShim to work with 0.23. Because the following casting does not work anymore. CombineFileSplit[] splits = (CombineFileSplit[]) super.getSplits(job, numSplits); And the paths of hadoop jars build.properties should be changed according to the 0.23 tar ball
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-17 07:33:27, Carl Steinbach wrote:

          > trunk/build.properties, line 13

          > <https://reviews.apache.org/r/2762/diff/1/?file=56790#file56790line13>

          >

          > For the time being we need to stick with 0.20.1 as the default Hadoop version against which Hive is tested, but we should start using 0.23.0 for the security tests, and should ensure that all tests pass when run against 0.23.0.

          >

          > Please set hadoop.version to 0.20.1 and hadoop.security.version to 0.23.0, and ensure that all tests pass when run as "ant test -Dhadoop.version=0.23.0".

          >

          > Also, I added 0.23.0 to the cloudera and facebook hive-deps mirrors, so please reference those locations instead.

          Thanks for taking a look Carl.

          I agree that we should leave 0.20.1 as the default. I think security tests should be left at the default for now too. This issue was to get Hive compiling against Hadoop 0.23.0, and I imagined any unit test failures against 0.23.0 could be fixed in follow up issues. Does that sound reasonable?

          • Tom

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2762/#review3316
          -----------------------------------------------------------

          On 2011-11-07 23:55:47, Tom White wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2762/

          -----------------------------------------------------------

          (Updated 2011-11-07 23:55:47)

          Review request for hive.

          Summary

          -------

          See https://issues.apache.org/jira/browse/HIVE-2468

          This addresses bug HIVE-2468.

          https://issues.apache.org/jira/browse/HIVE-2468

          Diffs

          -----

          trunk/build.properties 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775

          trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775

          trunk/shims/build.xml 1196775

          trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775

          trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775

          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION

          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775

          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775

          Diff: https://reviews.apache.org/r/2762/diff

          Testing

          -------

          Thanks,

          Tom

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-17 07:33:27, Carl Steinbach wrote: > trunk/build.properties, line 13 > < https://reviews.apache.org/r/2762/diff/1/?file=56790#file56790line13 > > > For the time being we need to stick with 0.20.1 as the default Hadoop version against which Hive is tested, but we should start using 0.23.0 for the security tests, and should ensure that all tests pass when run against 0.23.0. > > Please set hadoop.version to 0.20.1 and hadoop.security.version to 0.23.0, and ensure that all tests pass when run as "ant test -Dhadoop.version=0.23.0". > > Also, I added 0.23.0 to the cloudera and facebook hive-deps mirrors, so please reference those locations instead. Thanks for taking a look Carl. I agree that we should leave 0.20.1 as the default. I think security tests should be left at the default for now too. This issue was to get Hive compiling against Hadoop 0.23.0, and I imagined any unit test failures against 0.23.0 could be fixed in follow up issues. Does that sound reasonable? Tom ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/#review3316 ----------------------------------------------------------- On 2011-11-07 23:55:47, Tom White wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ ----------------------------------------------------------- (Updated 2011-11-07 23:55:47) Review request for hive. Summary ------- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468 . https://issues.apache.org/jira/browse/HIVE-2468 Diffs ----- trunk/build.properties 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775 trunk/shims/build.xml 1196775 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775 trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775 trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775 Diff: https://reviews.apache.org/r/2762/diff Testing ------- Thanks, Tom
          Hide
          jiraposter@reviews.apache.org added a comment -

          On 2011-11-17 07:33:27, Carl Steinbach wrote:

          > trunk/build.properties, line 13

          > <https://reviews.apache.org/r/2762/diff/1/?file=56790#file56790line13>

          >

          > For the time being we need to stick with 0.20.1 as the default Hadoop version against which Hive is tested, but we should start using 0.23.0 for the security tests, and should ensure that all tests pass when run against 0.23.0.

          >

          > Please set hadoop.version to 0.20.1 and hadoop.security.version to 0.23.0, and ensure that all tests pass when run as "ant test -Dhadoop.version=0.23.0".

          >

          > Also, I added 0.23.0 to the cloudera and facebook hive-deps mirrors, so please reference those locations instead.

          Tom White wrote:

          Thanks for taking a look Carl.

          I agree that we should leave 0.20.1 as the default. I think security tests should be left at the default for now too. This issue was to get Hive compiling against Hadoop 0.23.0, and I imagined any unit test failures against 0.23.0 could be fixed in follow up issues. Does that sound reasonable?

          Sounds good to me.

          • Carl

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2762/#review3316
          -----------------------------------------------------------

          On 2011-11-07 23:55:47, Tom White wrote:

          -----------------------------------------------------------

          This is an automatically generated e-mail. To reply, visit:

          https://reviews.apache.org/r/2762/

          -----------------------------------------------------------

          (Updated 2011-11-07 23:55:47)

          Review request for hive.

          Summary

          -------

          See https://issues.apache.org/jira/browse/HIVE-2468

          This addresses bug HIVE-2468.

          https://issues.apache.org/jira/browse/HIVE-2468

          Diffs

          -----

          trunk/build.properties 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775

          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775

          trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775

          trunk/shims/build.xml 1196775

          trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775

          trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775

          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION

          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION

          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775

          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775

          Diff: https://reviews.apache.org/r/2762/diff

          Testing

          -------

          Thanks,

          Tom

          Show
          jiraposter@reviews.apache.org added a comment - On 2011-11-17 07:33:27, Carl Steinbach wrote: > trunk/build.properties, line 13 > < https://reviews.apache.org/r/2762/diff/1/?file=56790#file56790line13 > > > For the time being we need to stick with 0.20.1 as the default Hadoop version against which Hive is tested, but we should start using 0.23.0 for the security tests, and should ensure that all tests pass when run against 0.23.0. > > Please set hadoop.version to 0.20.1 and hadoop.security.version to 0.23.0, and ensure that all tests pass when run as "ant test -Dhadoop.version=0.23.0". > > Also, I added 0.23.0 to the cloudera and facebook hive-deps mirrors, so please reference those locations instead. Tom White wrote: Thanks for taking a look Carl. I agree that we should leave 0.20.1 as the default. I think security tests should be left at the default for now too. This issue was to get Hive compiling against Hadoop 0.23.0, and I imagined any unit test failures against 0.23.0 could be fixed in follow up issues. Does that sound reasonable? Sounds good to me. Carl ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/#review3316 ----------------------------------------------------------- On 2011-11-07 23:55:47, Tom White wrote: ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ ----------------------------------------------------------- (Updated 2011-11-07 23:55:47) Review request for hive. Summary ------- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468 . https://issues.apache.org/jira/browse/HIVE-2468 Diffs ----- trunk/build.properties 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1196775 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1196775 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1196775 trunk/shims/build.xml 1196775 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1196775 trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1196775 trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/EmptyShim.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/HiveHarFileSystem.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSecretManager.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1196775 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1196775 Diff: https://reviews.apache.org/r/2762/diff Testing ------- Thanks, Tom
          Hide
          Thomas Weise added a comment -

          The proposed change would duplicate package/class names from 20S shim. That's problematic as all go into a single shim jar file. I would suggest to use a different package (or rename all classes) for the .23 shim. In those cases where classes from 20S are copied w/o modification, we should try to separate them out to a common location that is shared between 20S and 23?

          Show
          Thomas Weise added a comment - The proposed change would duplicate package/class names from 20S shim. That's problematic as all go into a single shim jar file. I would suggest to use a different package (or rename all classes) for the .23 shim. In those cases where classes from 20S are copied w/o modification, we should try to separate them out to a common location that is shared between 20S and 23?
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2762/
          -----------------------------------------------------------

          (Updated 2011-11-18 22:06:29.024823)

          Review request for hive.

          Changes
          -------

          Thanks for all the feedback! Here's a new patch which addresses it.

          • Change build.properties to the 0.20-based version numbers (build against 23 with "ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23")
          • Makes the change to bin/hive that Amareshwari suggested.
          • Removed duplicate classes names from the 0.23 shim as Thomas suggested. I also removed some classes that I needlessly introduced into the 23 shim tree in the last patch.

          Amareshwari, can you post your changes to CombineFileInputFormatShim so we can include those too?

          I also need to run unit tests against 0.20 to check there are no regressions.

          Summary
          -------

          See https://issues.apache.org/jira/browse/HIVE-2468

          This addresses bug HIVE-2468.
          https://issues.apache.org/jira/browse/HIVE-2468

          Diffs (updated)


          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION
          trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1203794
          trunk/shims/build.xml 1203794
          trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1203794
          trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1203794
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1203794
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1203794
          trunk/build.properties 1203794
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1203794
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java PRE-CREATION
          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1203794
          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1203794

          Diff: https://reviews.apache.org/r/2762/diff

          Testing
          -------

          Thanks,

          Tom

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ ----------------------------------------------------------- (Updated 2011-11-18 22:06:29.024823) Review request for hive. Changes ------- Thanks for all the feedback! Here's a new patch which addresses it. Change build.properties to the 0.20-based version numbers (build against 23 with "ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23") Makes the change to bin/hive that Amareshwari suggested. Removed duplicate classes names from the 0.23 shim as Thomas suggested. I also removed some classes that I needlessly introduced into the 23 shim tree in the last patch. Amareshwari, can you post your changes to CombineFileInputFormatShim so we can include those too? I also need to run unit tests against 0.20 to check there are no regressions. Summary ------- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468 . https://issues.apache.org/jira/browse/HIVE-2468 Diffs (updated) trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1203794 trunk/shims/build.xml 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1203794 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1203794 trunk/build.properties 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1203794 trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1203794 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1203794 Diff: https://reviews.apache.org/r/2762/diff Testing ------- Thanks, Tom
          Hide
          Tom White added a comment -

          All tests pass against 0.20.

          Show
          Tom White added a comment - All tests pass against 0.20.
          Hide
          Carl Steinbach added a comment -

          @Tom: The latest patch on RB looks good. Can you please attach it here and click the box that gives license rights to the ASF? Thanks!

          Show
          Carl Steinbach added a comment - @Tom: The latest patch on RB looks good. Can you please attach it here and click the box that gives license rights to the ASF? Thanks!
          Hide
          Amareshwari Sriramadasu added a comment -

          I tried to make combine file input format, but
          ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23 is giving errors.

          I modified build.properties to pickup jars from new locations in hadoop 0.23 tarball. Still I'm getting some errors. For some reason it was not able to find MiniDFSCluster and HadoopArchives classes. Also, it could not find jetty libraries for Jetty23Shims. Noticed that hadoop 0.23 tarball does not have jetty jars in lib/.
          Tom, is it some other way you could compile against 0.23?

          Show
          Amareshwari Sriramadasu added a comment - I tried to make combine file input format, but ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23 is giving errors. I modified build.properties to pickup jars from new locations in hadoop 0.23 tarball. Still I'm getting some errors. For some reason it was not able to find MiniDFSCluster and HadoopArchives classes. Also, it could not find jetty libraries for Jetty23Shims. Noticed that hadoop 0.23 tarball does not have jetty jars in lib/. Tom, is it some other way you could compile against 0.23?
          Hide
          Tom White added a comment -

          Thanks for the report, Amareshwari. I cleared my Ivy cache and hit the same problem (it was compiling fine against 0.23.0 before). I'll create a new patch.

          Show
          Tom White added a comment - Thanks for the report, Amareshwari. I cleared my Ivy cache and hit the same problem (it was compiling fine against 0.23.0 before). I'll create a new patch.
          Hide
          Tom White added a comment -

          It's not complete yet, but here's an updated patch. I still need to fix shim tests, and the missing ProxyLocalFileSystem in 0.23.

          Show
          Tom White added a comment - It's not complete yet, but here's an updated patch. I still need to fix shim tests, and the missing ProxyLocalFileSystem in 0.23.
          Hide
          Tom White added a comment -

          Here's a work in progress patch which passes some of the tests now.

          • Moved ProxyLocalFileSystem and ProxyFileSystem to common shims.
          • HADOOP-2081 was causing problems with hive.metastore.event.expiry.duration and hive.metastore.event.clean.freq, which were set to 0L, so I changed these to 0.
          • org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat is (still) failing due to the issue Amareshwari pointed out.
          • Some tests are failing due to class not found issues, which are due to incorrect classpaths. These still need fixing - I'm running some more tests to see if I can isolate the cause.
          Show
          Tom White added a comment - Here's a work in progress patch which passes some of the tests now. Moved ProxyLocalFileSystem and ProxyFileSystem to common shims. HADOOP-2081 was causing problems with hive.metastore.event.expiry.duration and hive.metastore.event.clean.freq, which were set to 0L, so I changed these to 0. org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat is (still) failing due to the issue Amareshwari pointed out. Some tests are failing due to class not found issues, which are due to incorrect classpaths. These still need fixing - I'm running some more tests to see if I can isolate the cause.
          Hide
          Tom White added a comment -

          This patch fixes org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat (the CombineFileSplit problem), and the classpath problems. I ran the whole unit test suite against Hadoop 0.23 and over 95% of the tests pass now.

          Show
          Tom White added a comment - This patch fixes org.apache.hadoop.hive.ql.io.TestSymlinkTextInputFormat (the CombineFileSplit problem), and the classpath problems. I ran the whole unit test suite against Hadoop 0.23 and over 95% of the tests pass now.
          Hide
          Amareshwari Sriramadasu added a comment -

          I could compile hive against Hadoop 0.23 without any issues with the latest patch. Thanks Tom.
          Can you also include the change in bin/hive and make it Patch available?

          I also started running tests. Will get back if i find any issues.

          Show
          Amareshwari Sriramadasu added a comment - I could compile hive against Hadoop 0.23 without any issues with the latest patch. Thanks Tom. Can you also include the change in bin/hive and make it Patch available? I also started running tests. Will get back if i find any issues.
          Hide
          Tom White added a comment -

          New patch with Amareshwari's change to bin/hive.

          Show
          Tom White added a comment - New patch with Amareshwari's change to bin/hive.
          Hide
          jiraposter@reviews.apache.org added a comment -

          -----------------------------------------------------------
          This is an automatically generated e-mail. To reply, visit:
          https://reviews.apache.org/r/2762/
          -----------------------------------------------------------

          (Updated 2011-11-30 16:57:27.344021)

          Review request for hive.

          Summary
          -------

          See https://issues.apache.org/jira/browse/HIVE-2468

          This addresses bug HIVE-2468.
          https://issues.apache.org/jira/browse/HIVE-2468

          Diffs (updated)


          trunk/bin/hive 1203794
          trunk/build-common.xml 1203794
          trunk/build.properties 1203794
          trunk/build.xml 1203794
          trunk/conf/hive-default.xml 1203794
          trunk/contrib/build.xml 1203794
          trunk/hbase-handler/build.xml 1203794
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1203794
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1203794
          trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1203794
          trunk/hwi/build.xml 1203794
          trunk/jdbc/build.xml 1203794
          trunk/ql/build.xml 1203794
          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1203794
          trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobTrackerURLResolver.java 1203794
          trunk/service/build.xml 1203794
          trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1203794
          trunk/shims/build.xml 1203794
          trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyFileSystem.java 1203794
          trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java 1203794
          trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1203794
          trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1203794
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java PRE-CREATION
          trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java PRE-CREATION
          trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION
          trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION
          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1203794
          trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1203794

          Diff: https://reviews.apache.org/r/2762/diff

          Testing
          -------

          Thanks,

          Tom

          Show
          jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/2762/ ----------------------------------------------------------- (Updated 2011-11-30 16:57:27.344021) Review request for hive. Summary ------- See https://issues.apache.org/jira/browse/HIVE-2468 This addresses bug HIVE-2468 . https://issues.apache.org/jira/browse/HIVE-2468 Diffs (updated) trunk/bin/hive 1203794 trunk/build-common.xml 1203794 trunk/build.properties 1203794 trunk/build.xml 1203794 trunk/conf/hive-default.xml 1203794 trunk/contrib/build.xml 1203794 trunk/hbase-handler/build.xml 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java 1203794 trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java 1203794 trunk/hwi/build.xml 1203794 trunk/jdbc/build.xml 1203794 trunk/ql/build.xml 1203794 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1203794 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobTrackerURLResolver.java 1203794 trunk/service/build.xml 1203794 trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1203794 trunk/shims/build.xml 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyFileSystem.java 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java 1203794 trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java 1203794 trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java 1203794 trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java PRE-CREATION trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyFileSystem.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java PRE-CREATION trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java 1203794 trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java 1203794 Diff: https://reviews.apache.org/r/2762/diff Testing ------- Thanks, Tom
          Hide
          Thomas Weise added a comment -

          I assume (based on findings in linked Pig issue) that when Hive is built against .23 then it won't be byte code compatible with .20 and so even though the shim is present cannot run on .20?

          Show
          Thomas Weise added a comment - I assume (based on findings in linked Pig issue) that when Hive is built against .23 then it won't be byte code compatible with .20 and so even though the shim is present cannot run on .20?
          Hide
          Carl Steinbach added a comment -

          +1. Will commit if tests pass.

          Show
          Carl Steinbach added a comment - +1. Will commit if tests pass.
          Hide
          Carl Steinbach added a comment -

          Committed to trunk. Thanks Tom!

          Show
          Carl Steinbach added a comment - Committed to trunk. Thanks Tom!
          Hide
          Carl Steinbach added a comment -

          I'm going to backport this patch to 0.8.0. Reassigning to myself until that work is finished.

          Also, I created a new Jenkins job here: https://builds.apache.org/job/Hive-trunk-h0.23.0/

          Show
          Carl Steinbach added a comment - I'm going to backport this patch to 0.8.0. Reassigning to myself until that work is finished. Also, I created a new Jenkins job here: https://builds.apache.org/job/Hive-trunk-h0.23.0/
          Hide
          Hudson added a comment -

          Integrated in Hive-trunk-h0.21 #1115 (See https://builds.apache.org/job/Hive-trunk-h0.21/1115/)
          HIVE-2468. Make Hive compile against Hadoop 0.23 (Tom White via cws)

          cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208940
          Files :

          • /hive/trunk/bin/hive
          • /hive/trunk/build-common.xml
          • /hive/trunk/build.properties
          • /hive/trunk/build.xml
          • /hive/trunk/conf/hive-default.xml
          • /hive/trunk/contrib/build.xml
          • /hive/trunk/hbase-handler/build.xml
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java
          • /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java
          • /hive/trunk/hwi/build.xml
          • /hive/trunk/jdbc/build.xml
          • /hive/trunk/ql/build.xml
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java
          • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobTrackerURLResolver.java
          • /hive/trunk/service/build.xml
          • /hive/trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java
          • /hive/trunk/shims/build.xml
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyFileSystem.java
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java
          • /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java
          • /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java
          • /hive/trunk/shims/src/0.23
          • /hive/trunk/shims/src/0.23/java
          • /hive/trunk/shims/src/0.23/java/org
          • /hive/trunk/shims/src/0.23/java/org/apache
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java
          • /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/fs
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyFileSystem.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java
          • /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java
          Show
          Hudson added a comment - Integrated in Hive-trunk-h0.21 #1115 (See https://builds.apache.org/job/Hive-trunk-h0.21/1115/ ) HIVE-2468 . Make Hive compile against Hadoop 0.23 (Tom White via cws) cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1208940 Files : /hive/trunk/bin/hive /hive/trunk/build-common.xml /hive/trunk/build.properties /hive/trunk/build.xml /hive/trunk/conf/hive-default.xml /hive/trunk/contrib/build.xml /hive/trunk/hbase-handler/build.xml /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableOutputFormat.java /hive/trunk/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHFileOutputFormat.java /hive/trunk/hwi/build.xml /hive/trunk/jdbc/build.xml /hive/trunk/ql/build.xml /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/JobTrackerURLResolver.java /hive/trunk/service/build.xml /hive/trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java /hive/trunk/shims/build.xml /hive/trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyFileSystem.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java /hive/trunk/shims/src/0.20/java/org/apache/hadoop/hive/shims/Hadoop20Shims.java /hive/trunk/shims/src/0.20S/java/org/apache/hadoop/hive/shims/Hadoop20SShims.java /hive/trunk/shims/src/0.23 /hive/trunk/shims/src/0.23/java /hive/trunk/shims/src/0.23/java/org /hive/trunk/shims/src/0.23/java/org/apache /hive/trunk/shims/src/0.23/java/org/apache/hadoop /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Hadoop23Shims.java /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/shims/Jetty23Shims.java /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenIdentifier23.java /hive/trunk/shims/src/0.23/java/org/apache/hadoop/hive/thrift/DelegationTokenSelector23.java /hive/trunk/shims/src/common/java/org/apache/hadoop/fs /hive/trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyFileSystem.java /hive/trunk/shims/src/common/java/org/apache/hadoop/fs/ProxyLocalFileSystem.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/HadoopShims.java /hive/trunk/shims/src/common/java/org/apache/hadoop/hive/shims/ShimLoader.java
          Hide
          Ahmed Radwan added a comment -

          I checked out trunk, then made the release tarball using ant binary. Tried to run against a real hadoop 0.23 cluster, but I am getting the following error when trying to start hive:

          Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims
          	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:277)
          	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:603)
          	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554)
          	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          	at java.lang.reflect.Method.invoke(Method.java:597)
          	at org.apache.hadoop.util.RunJar.main(RunJar.java:189)
          Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims
          	at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:225)
          	at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:270)
          	... 7 more
          Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims
          	at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:53)
          	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70)
          	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125)
          	at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:222)
          	... 8 more
          Caused by: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims
          	at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:102)
          	at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:94)
          	at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:65)
          	at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51)
          	... 11 more
          Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.shims.Hadoop23Shims
          	at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
          	at java.security.AccessController.doPrivileged(Native Method)
          	at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
          	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
          	at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
          	at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
          	at java.lang.Class.forName0(Native Method)
          	at java.lang.Class.forName(Class.java:169)
          	at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:99)
          	... 14 more
          
          Show
          Ahmed Radwan added a comment - I checked out trunk, then made the release tarball using ant binary. Tried to run against a real hadoop 0.23 cluster, but I am getting the following error when trying to start hive: Exception in thread "main" java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:277) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:603) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:554) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:189) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:225) at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:270) ... 7 more Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:53) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:70) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:125) at org.apache.hadoop.hive.ql.metadata.HiveUtils.getAuthenticator(HiveUtils.java:222) ... 8 more Caused by: java.lang.RuntimeException: Could not load shims in class org.apache.hadoop.hive.shims.Hadoop23Shims at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:102) at org.apache.hadoop.hive.shims.ShimLoader.loadShims(ShimLoader.java:94) at org.apache.hadoop.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:65) at org.apache.hadoop.hive.ql.security.HadoopDefaultAuthenticator.setConf(HadoopDefaultAuthenticator.java:51) ... 11 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.shims.Hadoop23Shims at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang. ClassLoader .loadClass( ClassLoader .java:307) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang. ClassLoader .loadClass( ClassLoader .java:252) at java.lang. ClassLoader .loadClassInternal( ClassLoader .java:320) at java.lang. Class .forName0(Native Method) at java.lang. Class .forName( Class .java:169) at org.apache.hadoop.hive.shims.ShimLoader.createShim(ShimLoader.java:99) ... 14 more
          Hide
          Carl Steinbach added a comment -

          @Ahmed: Please try recompiling Hive with the following build properties:

          ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23

          If this works then we need to investigate ways of building a Hive binary that works with both 0.20.1 and 0.23.0. Otherwise there's a more serious bug that we need to investigate. Thanks.

          Show
          Carl Steinbach added a comment - @Ahmed: Please try recompiling Hive with the following build properties: ant -Dhadoop.version=0.23.0 -Dhadoop.security.version=0.23.0 -Dhadoop.security.version.prefix=0.23 If this works then we need to investigate ways of building a Hive binary that works with both 0.20.1 and 0.23.0. Otherwise there's a more serious bug that we need to investigate. Thanks.
          Hide
          Ahmed Radwan added a comment -

          Used these flags in recompiling. Rerunning against 0.23, the previous runtime exception seem to disappear. Thanks!

          Show
          Ahmed Radwan added a comment - Used these flags in recompiling. Rerunning against 0.23, the previous runtime exception seem to disappear. Thanks!
          Hide
          Namit Jain added a comment -

          @Carl: can the above be used to compile hive source code also ?

          Show
          Namit Jain added a comment - @Carl: can the above be used to compile hive source code also ?

            People

            • Assignee:
              Tom White
              Reporter:
              Konstantin Shvachko
            • Votes:
              0 Vote for this issue
              Watchers:
              15 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development