Hive
  1. Hive
  2. HIVE-3156

Support MiniMR tests for Hadoop-0.23

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 0.9.1
    • Fix Version/s: None
    • Component/s: Tests
    • Labels:
      None
    • Environment:

      Hive-0.9.1 + Hadoop-0.23.0

      Description

      When MiniMR tests are ran with hadoop-0.23.0, tests fails throwing following exception.

      java.lang.UnsupportedOperationException
      at org.apache.hadoop.mapred.MiniMRCluster.getJobTrackerPort(MiniMRCluster.java:88)
      at org.apache.hadoop.hive.ql.QTestUtil.initConf(QTestUtil.java:215)

      In hadoop-0.23 ,few mapreduce API's are not supported like getJobTrackerPort().

      Here, we need to use API's based on hadoop versions.

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3341 [jira] Making hive tests run against different MR versions
        (Sushanth Sowmyan via Ashutosh Chauhan)

        Summary:
        HIVE-3341 Making hive tests run against different MR versions

        After we build hive, we want to have the ability to run unit tests against specific hadoop versions. Currently, the classpath constructed has multiple hadoop jars, which makes compiling okay, but running non-deterministic.

        An example is HIVE-3156, where running against 0.23 shows issues with a couple of tests (which should either be shimmed out, or separated into directories the way it's done in the shims/ directory) - It would also be nice to find these issues out at test-compile time itself, rather than having them fail at test runtime.

        With this patch, we can set ant variable hadoop.mr.rev to 20, 20S or 23 to test against a particular version.

        Test Plan: current tests continue to work - this is a build system change

        Reviewers: JIRA, ashutoshc, cwsteinbach

        Reviewed By: ashutoshc

        Differential Revision: https://reviews.facebook.net/D4557 (Revision 1372745)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372745
        Files :

        • /hive/trunk/build-common.xml
        • /hive/trunk/build.properties
        • /hive/trunk/ivy/common-configurations.xml
        • /hive/trunk/shims/build.xml
        • /hive/trunk/shims/src/common-secure/test
        • /hive/trunk/shims/src/common-secure/test/org
        • /hive/trunk/shims/src/common-secure/test/org/apache
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
        • /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
        • /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3341 [jira] Making hive tests run against different MR versions (Sushanth Sowmyan via Ashutosh Chauhan) Summary: HIVE-3341 Making hive tests run against different MR versions After we build hive, we want to have the ability to run unit tests against specific hadoop versions. Currently, the classpath constructed has multiple hadoop jars, which makes compiling okay, but running non-deterministic. An example is HIVE-3156 , where running against 0.23 shows issues with a couple of tests (which should either be shimmed out, or separated into directories the way it's done in the shims/ directory) - It would also be nice to find these issues out at test-compile time itself, rather than having them fail at test runtime. With this patch, we can set ant variable hadoop.mr.rev to 20, 20S or 23 to test against a particular version. Test Plan: current tests continue to work - this is a build system change Reviewers: JIRA, ashutoshc, cwsteinbach Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D4557 (Revision 1372745) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372745 Files : /hive/trunk/build-common.xml /hive/trunk/build.properties /hive/trunk/ivy/common-configurations.xml /hive/trunk/shims/build.xml /hive/trunk/shims/src/common-secure/test /hive/trunk/shims/src/common-secure/test/org /hive/trunk/shims/src/common-secure/test/org/apache /hive/trunk/shims/src/common-secure/test/org/apache/hadoop /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1605 (See https://builds.apache.org/job/Hive-trunk-h0.21/1605/)
        HIVE-3341 [jira] Making hive tests run against different MR versions
        (Sushanth Sowmyan via Ashutosh Chauhan)

        Summary:
        HIVE-3341 Making hive tests run against different MR versions

        After we build hive, we want to have the ability to run unit tests against specific hadoop versions. Currently, the classpath constructed has multiple hadoop jars, which makes compiling okay, but running non-deterministic.

        An example is HIVE-3156, where running against 0.23 shows issues with a couple of tests (which should either be shimmed out, or separated into directories the way it's done in the shims/ directory) - It would also be nice to find these issues out at test-compile time itself, rather than having them fail at test runtime.

        With this patch, we can set ant variable hadoop.mr.rev to 20, 20S or 23 to test against a particular version.

        Test Plan: current tests continue to work - this is a build system change

        Reviewers: JIRA, ashutoshc, cwsteinbach

        Reviewed By: ashutoshc

        Differential Revision: https://reviews.facebook.net/D4557 (Revision 1372745)

        Result = SUCCESS
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372745
        Files :

        • /hive/trunk/build-common.xml
        • /hive/trunk/build.properties
        • /hive/trunk/ivy/common-configurations.xml
        • /hive/trunk/shims/build.xml
        • /hive/trunk/shims/src/common-secure/test
        • /hive/trunk/shims/src/common-secure/test/org
        • /hive/trunk/shims/src/common-secure/test/org/apache
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
        • /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
        • /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java
        • /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1605 (See https://builds.apache.org/job/Hive-trunk-h0.21/1605/ ) HIVE-3341 [jira] Making hive tests run against different MR versions (Sushanth Sowmyan via Ashutosh Chauhan) Summary: HIVE-3341 Making hive tests run against different MR versions After we build hive, we want to have the ability to run unit tests against specific hadoop versions. Currently, the classpath constructed has multiple hadoop jars, which makes compiling okay, but running non-deterministic. An example is HIVE-3156 , where running against 0.23 shows issues with a couple of tests (which should either be shimmed out, or separated into directories the way it's done in the shims/ directory) - It would also be nice to find these issues out at test-compile time itself, rather than having them fail at test runtime. With this patch, we can set ant variable hadoop.mr.rev to 20, 20S or 23 to test against a particular version. Test Plan: current tests continue to work - this is a build system change Reviewers: JIRA, ashutoshc, cwsteinbach Reviewed By: ashutoshc Differential Revision: https://reviews.facebook.net/D4557 (Revision 1372745) Result = SUCCESS hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372745 Files : /hive/trunk/build-common.xml /hive/trunk/build.properties /hive/trunk/ivy/common-configurations.xml /hive/trunk/shims/build.xml /hive/trunk/shims/src/common-secure/test /hive/trunk/shims/src/common-secure/test/org /hive/trunk/shims/src/common-secure/test/org/apache /hive/trunk/shims/src/common-secure/test/org/apache/hadoop /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java /hive/trunk/shims/src/common-secure/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestHadoop20SAuthBridge.java /hive/trunk/shims/src/test/org/apache/hadoop/hive/thrift/TestZooKeeperTokenStore.java

          People

          • Assignee:
            Unassigned
            Reporter:
            Rohith
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:

              Development