Hive
  1. Hive
  2. HIVE-3416

Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.10.0
    • Component/s: Build Infrastructure
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      TestAvroSerdeUtils determinSchemaCanReadSchemaFromHDFS is failing when running hive on hadoop23:

      $ant very-clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23

      $ant test -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23 -Dtestcase=TestAvroSerdeUtils

      <testcase classname="org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils" name="determineSchemaCanReadSchemaFromHDFS" time="0.21">
      <error message="org/apache/hadoop/net/StaticMapping" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: org/apache/hadoop/net/StaticMapping
      at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:534)
      at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:489)
      at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:360)
      at org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS(TestAvroSerdeUtils.java:187)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
      at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
      at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
      at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
      at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
      at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
      at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
      at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
      at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
      at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
      at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
      at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
      at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
      at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
      at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
      at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.StaticMapping
      at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
      ... 25 more
      </error>
      </testcase>

        Activity

        Hide
        Ashutosh Chauhan added a comment -

        This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.

        Show
        Ashutosh Chauhan added a comment - This issue is fixed and released as part of 0.10.0 release. If you find an issue which seems to be related to this one, please create a new jira and link this one with new jira.
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/)
        HIVE-3416 [jira] Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23
        (Zhenxiao Luo via Carl Steinbach)

        Summary:
        HIVE-3416: Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23

        TestAvroSerdeUtils determinSchemaCanReadSchemaFromHDFS is failing when running hive on hadoop23:

        $ant very-clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23

        $ant test -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23 -Dtestcase=TestAvroSerdeUtils

        <testcase classname="org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils" name="determineSchemaCanReadSchemaFromHDFS" time="0.21">
        <error message="org/apache/hadoop/net/StaticMapping" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: org/apache/hadoop/net/StaticMapping
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:534)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:489)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:360)
        at org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS(TestAvroSerdeUtils.java:187)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
        Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.StaticMapping
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
        ... 25 more
        </error>
        </testcase>

        Test Plan: EMPTY

        Reviewers: JIRA

        Differential Revision: https://reviews.facebook.net/D5025 (Revision 1380490)

        Result = ABORTED
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380490
        Files :

        • /hive/trunk/serde/ivy.xml
        • /hive/trunk/shims/ivy.xml
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #54 (See https://builds.apache.org/job/Hive-trunk-hadoop2/54/ ) HIVE-3416 [jira] Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23 (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3416 : Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23 TestAvroSerdeUtils determinSchemaCanReadSchemaFromHDFS is failing when running hive on hadoop23: $ant very-clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23 $ant test -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23 -Dtestcase=TestAvroSerdeUtils <testcase classname="org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils" name="determineSchemaCanReadSchemaFromHDFS" time="0.21"> <error message="org/apache/hadoop/net/StaticMapping" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: org/apache/hadoop/net/StaticMapping at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:534) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:489) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:360) at org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS(TestAvroSerdeUtils.java:187) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.StaticMapping at java.net.URLClassLoader$1.run(URLClassLoader.java:217) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang.ClassLoader.loadClass(ClassLoader.java:321) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) at java.lang.ClassLoader.loadClass(ClassLoader.java:266) ... 25 more </error> </testcase> Test Plan: EMPTY Reviewers: JIRA Differential Revision: https://reviews.facebook.net/D5025 (Revision 1380490) Result = ABORTED cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380490 Files : /hive/trunk/serde/ivy.xml /hive/trunk/shims/ivy.xml
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1646 (See https://builds.apache.org/job/Hive-trunk-h0.21/1646/)
        HIVE-3416 [jira] Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23
        (Zhenxiao Luo via Carl Steinbach)

        Summary:
        HIVE-3416: Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23

        TestAvroSerdeUtils determinSchemaCanReadSchemaFromHDFS is failing when running hive on hadoop23:

        $ant very-clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23

        $ant test -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23 -Dtestcase=TestAvroSerdeUtils

        <testcase classname="org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils" name="determineSchemaCanReadSchemaFromHDFS" time="0.21">
        <error message="org/apache/hadoop/net/StaticMapping" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: org/apache/hadoop/net/StaticMapping
        at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:534)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:489)
        at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:360)
        at org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS(TestAvroSerdeUtils.java:187)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:616)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20)
        at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49)
        at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191)
        at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:236)
        at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911)
        at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768)
        Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.StaticMapping
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
        ... 25 more
        </error>
        </testcase>

        Test Plan: EMPTY

        Reviewers: JIRA

        Differential Revision: https://reviews.facebook.net/D5025 (Revision 1380490)

        Result = FAILURE
        cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380490
        Files :

        • /hive/trunk/serde/ivy.xml
        • /hive/trunk/shims/ivy.xml
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1646 (See https://builds.apache.org/job/Hive-trunk-h0.21/1646/ ) HIVE-3416 [jira] Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23 (Zhenxiao Luo via Carl Steinbach) Summary: HIVE-3416 : Fix TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS when running Hive on hadoop23 TestAvroSerdeUtils determinSchemaCanReadSchemaFromHDFS is failing when running hive on hadoop23: $ant very-clean package -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23 $ant test -Dhadoop.version=0.23.1 -Dhadoop-0.23.version=0.23.1 -Dhadoop.mr.rev=23 -Dtestcase=TestAvroSerdeUtils <testcase classname="org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils" name="determineSchemaCanReadSchemaFromHDFS" time="0.21"> <error message="org/apache/hadoop/net/StaticMapping" type="java.lang.NoClassDefFoundError">java.lang.NoClassDefFoundError: org/apache/hadoop/net/StaticMapping at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:534) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:489) at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:360) at org.apache.hadoop.hive.serde2.avro.TestAvroSerdeUtils.determineSchemaCanReadSchemaFromHDFS(TestAvroSerdeUtils.java:187) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:616) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:44) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:41) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) at org.junit.runners.BlockJUnit4ClassRunner.runNotIgnored(BlockJUnit4ClassRunner.java:79) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:71) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:49) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:193) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:52) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:191) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:42) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:184) at org.junit.runners.ParentRunner.run(ParentRunner.java:236) at junit.framework.JUnit4TestAdapter.run(JUnit4TestAdapter.java:39) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.run(JUnitTestRunner.java:420) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.launch(JUnitTestRunner.java:911) at org.apache.tools.ant.taskdefs.optional.junit.JUnitTestRunner.main(JUnitTestRunner.java:768) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.net.StaticMapping at java.net.URLClassLoader$1.run(URLClassLoader.java:217) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang.ClassLoader.loadClass(ClassLoader.java:321) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) at java.lang.ClassLoader.loadClass(ClassLoader.java:266) ... 25 more </error> </testcase> Test Plan: EMPTY Reviewers: JIRA Differential Revision: https://reviews.facebook.net/D5025 (Revision 1380490) Result = FAILURE cws : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1380490 Files : /hive/trunk/serde/ivy.xml /hive/trunk/shims/ivy.xml
        Hide
        Carl Steinbach added a comment -

        Committed to trunk. Thanks Zhenxiao!

        Show
        Carl Steinbach added a comment - Committed to trunk. Thanks Zhenxiao!
        Hide
        Carl Steinbach added a comment -

        +1. Will commit if tests pass.

        Show
        Carl Steinbach added a comment - +1. Will commit if tests pass.
        Hide
        Zhenxiao Luo added a comment -

        Review request submitted at:
        https://reviews.facebook.net/D5025

        As stated in the previous comment, dependencies are added in shims/ivy.xml, so that they are included in test.classpath. They are also added in serde/ivy.xml, so that we have the note that serde has these dependencies.

        Show
        Zhenxiao Luo added a comment - Review request submitted at: https://reviews.facebook.net/D5025 As stated in the previous comment, dependencies are added in shims/ivy.xml, so that they are included in test.classpath. They are also added in serde/ivy.xml, so that we have the note that serde has these dependencies.
        Hide
        Zhenxiao Luo added a comment -

        The problem is, TestAvroSerdeUtils has a dependency on hadoop-hdfs-tests.jar in calling MiniDFSCluster.java, which has a dependency on hadoop-common-tests.jar in calling StaticMapping.java.

        These two dependency is not specified in serde/ivy.xml, and not included in hadoop23.test:
        hadoop-hdfs-*-tests.jar
        hadoop-common-*-tests.jar

        Just adding hadoop-hdfs-tests.jar does not implicitly add hadoop-common-tests.jar, I propose to add both dependencies, and also updated serde/ivy.xml first line in defining xml schema for maven, so that m:classifier could be used.

        Also, in build-common.xml, the following lines are in test.classpath definition:
        <fileset dir="$

        {build.ivy.lib.dir}

        /hadoop0.$

        {hadoop.mr.rev}

        .shim" includes="*.jar" erroronmissingdir="false" />
        <pathelement location="$

        {build.classes}

        " />

        <!-- we strip out hadoop jars present in places other than the hadoop shimmed dir-->
        <fileset dir="$

        {hive.root}/build/ivy/lib/test" includes="*.jar" erroronmissingdir="false"
        excludes="*/hive_.jar,*/hive-.jar,*/hadoop-.jar"/>
        <fileset dir="${hive.root}

        /build/ivy/lib/default" includes="*.jar" erroronmissingdir="false"
        excludes="*/hive_.jar,*/hive-.jar,*/hadoop-.jar" />

        Which excludes all the hadoop-*jars in build/ivy/lib/test directory from the test.classpath.

        So, adding the dependencies in serde/ivy.xml does not work. We need to add them in shims/ivy.xml, which is included in the test.classpath.

        Show
        Zhenxiao Luo added a comment - The problem is, TestAvroSerdeUtils has a dependency on hadoop-hdfs- tests.jar in calling MiniDFSCluster.java, which has a dependency on hadoop-common -tests.jar in calling StaticMapping.java. These two dependency is not specified in serde/ivy.xml, and not included in hadoop23.test: hadoop-hdfs-*-tests.jar hadoop-common-*-tests.jar Just adding hadoop-hdfs- tests.jar does not implicitly add hadoop-common -tests.jar, I propose to add both dependencies, and also updated serde/ivy.xml first line in defining xml schema for maven, so that m:classifier could be used. Also, in build-common.xml, the following lines are in test.classpath definition: <fileset dir="$ {build.ivy.lib.dir} /hadoop0.$ {hadoop.mr.rev} .shim" includes="*.jar" erroronmissingdir="false" /> <pathelement location="$ {build.classes} " /> <!-- we strip out hadoop jars present in places other than the hadoop shimmed dir--> <fileset dir="$ {hive.root}/build/ivy/lib/test" includes="*.jar" erroronmissingdir="false" excludes="* /hive_ .jar,* /hive- .jar,* /hadoop- .jar"/> <fileset dir="${hive.root} /build/ivy/lib/default" includes="*.jar" erroronmissingdir="false" excludes="* /hive_ .jar,* /hive- .jar,* /hadoop- .jar" /> Which excludes all the hadoop-*jars in build/ivy/lib/test directory from the test.classpath. So, adding the dependencies in serde/ivy.xml does not work. We need to add them in shims/ivy.xml, which is included in the test.classpath.

          People

          • Assignee:
            Zhenxiao Luo
            Reporter:
            Zhenxiao Luo
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development