Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-1612

Cannot build hive for hadoop 0.21.0

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.7.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Current trunk for 0.7.0 does not support building HIVE against the Hadoop 0.21.0 release.

      1. HIVE-1612.patch
        40 kB
        Daisuke Fujiwara
      2. HIVE-1612-for-r1043843.patch
        37 kB
        Bochun Bai
      3. HIVE-1612-for-r1043843.patch
        7 kB
        Bochun Bai
      4. HIVE-1612-for-r1127826.patch
        11 kB
        Bochun Bai

        Activity

        Hide
        bharathv bharath v added a comment -

        Any one working on this issue ?

        Show
        bharathv bharath v added a comment - Any one working on this issue ?
        Hide
        ajpahl1008 AJ Pahl added a comment -

        No one has reached out to me for further info or clarification.

        Sent via PocketJIRA - www.pocketjira.com

        Show
        ajpahl1008 AJ Pahl added a comment - No one has reached out to me for further info or clarification. Sent via PocketJIRA - www.pocketjira.com
        Hide
        appodictic Edward Capriolo added a comment -

        Their are issues open to get hive working with hadoop security. Since security is only in the 0.21 branch, thus someone is working on this.

        Show
        appodictic Edward Capriolo added a comment - Their are issues open to get hive working with hadoop security. Since security is only in the 0.21 branch, thus someone is working on this.
        Hide
        tlipcon Todd Lipcon added a comment -

        Currently the people working on security are doing so against features backported to 0.20S (see HIVE-1264 shims for example). This is distinct from 0.21, which only has a partial implementation of security features.

        Show
        tlipcon Todd Lipcon added a comment - Currently the people working on security are doing so against features backported to 0.20S (see HIVE-1264 shims for example). This is distinct from 0.21, which only has a partial implementation of security features.
        Hide
        dfujiwara Daisuke Fujiwara added a comment -

        Hi,
        I have managed to build Hive (trunk -r 1000628) against Hadoop 0.21.0.

        Following is what I had to do to make it work.
        1. build Hive as instructed in Hive wiki (ant pacakge).
        2. Copy hadoop-0.21.0.tar.gz to $

        {HIVE_DIR}

        /build/hadoopcore, untar it, and create/touch the "hadoop-0.21.0.installed" file
        3. Apply the patch
        4. build Hive again by issuing ant package "-Dhadoop.version=0.21.0 -Doffline=true"

        I understand that the patch and process to build is not in the pristine forms, but I would like to get some feedback and see if I can pursue this further.

        Thanks.

        Show
        dfujiwara Daisuke Fujiwara added a comment - Hi, I have managed to build Hive (trunk -r 1000628) against Hadoop 0.21.0. Following is what I had to do to make it work. 1. build Hive as instructed in Hive wiki (ant pacakge). 2. Copy hadoop-0.21.0.tar.gz to $ {HIVE_DIR} /build/hadoopcore, untar it, and create/touch the "hadoop-0.21.0.installed" file 3. Apply the patch 4. build Hive again by issuing ant package "-Dhadoop.version=0.21.0 -Doffline=true" I understand that the patch and process to build is not in the pristine forms, but I would like to get some feedback and see if I can pursue this further. Thanks.
        Hide
        singochina@gmail.com SingoWong added a comment -

        Hi,

        Can you execute the hive-ql in hive client? i got this error below:

        $ ./hive
        Exception in thread "main" java.lang.RuntimeException: Could not load shims
        in class null
        at org.apache.hadoop
        .hive.shims.ShimLoader.loadShims(ShimLoader.java:86)
        at org.apache.hadoop
        .hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:62)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:234)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
        sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
        Caused by: java.lang.NullPointerException
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:169)
        at org.apache.hadoop
        .hive.shims.ShimLoader.loadShims(ShimLoader.java:83)
        ... 7 more

        and cannot execute hive, is it HIVE dont support hadoop0.21.0 now?

        Thanks & Regards,
        Singo

        Show
        singochina@gmail.com SingoWong added a comment - Hi, Can you execute the hive-ql in hive client? i got this error below: $ ./hive Exception in thread "main" java.lang.RuntimeException: Could not load shims in class null at org.apache.hadoop .hive.shims.ShimLoader.loadShims(ShimLoader.java:86) at org.apache.hadoop .hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:62) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:234) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:192) Caused by: java.lang.NullPointerException at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:169) at org.apache.hadoop .hive.shims.ShimLoader.loadShims(ShimLoader.java:83) ... 7 more and cannot execute hive, is it HIVE dont support hadoop0.21.0 now? Thanks & Regards, Singo
        Hide
        ajpahl1008 AJ Pahl added a comment -

        As per Daisuke's instructions I was able to complile trunk-1000628 against Hadoop 0.21.0.
        Unfortunately, execution isn't exactly like Singo's but is still contains a few exceptions:

        -bash-3.2$ ./hive
        Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:185)
        Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
        ... 3 more

        Also starting hiveserver...

        Starting Hive Thrift Server
        Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/protocol/TProtocolFactory
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:247)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:185)
        Caused by: java.lang.ClassNotFoundException: org.apache.thrift.protocol.TProtocolFactory
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
        ... 3 more

        I've had success with compiling previous releases of 0.6.0 and 0.7.0 (prior to 0.21.0) so I'm
        pretty sure my environment is correct for testing this. (ie: Classpath etc)

        Daisuke, keep up the good work. I think this JIRA will be renamed DAISUKE-1612 if you can get it worked out.

        Show
        ajpahl1008 AJ Pahl added a comment - As per Daisuke's instructions I was able to complile trunk-1000628 against Hadoop 0.21.0. Unfortunately, execution isn't exactly like Singo's but is still contains a few exceptions: -bash-3.2$ ./hive Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:185) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) ... 3 more Also starting hiveserver... Starting Hive Thrift Server Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/protocol/TProtocolFactory at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:247) at org.apache.hadoop.util.RunJar.main(RunJar.java:185) Caused by: java.lang.ClassNotFoundException: org.apache.thrift.protocol.TProtocolFactory at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:307) at java.lang.ClassLoader.loadClass(ClassLoader.java:252) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320) ... 3 more I've had success with compiling previous releases of 0.6.0 and 0.7.0 (prior to 0.21.0) so I'm pretty sure my environment is correct for testing this. (ie: Classpath etc) Daisuke, keep up the good work. I think this JIRA will be renamed DAISUKE-1612 if you can get it worked out.
        Hide
        daisuke.fujiwara Daisuke Fujiwara added a comment -

        Hi AJ,

        I would like to get more information about your environment because I can successfully start the hive cli session as well as hive thrift server, and they seem to work fine.

        Your error messages suggest to me that your class path is not set correctly for some reason.
        However those two classes that are reported (org/apache/hadoop/hive/conf/HiveConf and org/apache/thrift/protocol/TProtocolFactory) are present in the $HIVE_HOME/lib folder, namely hive-common-0.7.0.jar and libthrift.jar. Delving further into how the classpath is set, it seems like bin/hive takes care of it.
        So I am not sure why your instance fails to locate those classes.

        Please let me know if you have any suggestions.

        Show
        daisuke.fujiwara Daisuke Fujiwara added a comment - Hi AJ, I would like to get more information about your environment because I can successfully start the hive cli session as well as hive thrift server, and they seem to work fine. Your error messages suggest to me that your class path is not set correctly for some reason. However those two classes that are reported (org/apache/hadoop/hive/conf/HiveConf and org/apache/thrift/protocol/TProtocolFactory) are present in the $HIVE_HOME/lib folder, namely hive-common-0.7.0.jar and libthrift.jar. Delving further into how the classpath is set, it seems like bin/hive takes care of it. So I am not sure why your instance fails to locate those classes. Please let me know if you have any suggestions.
        Hide
        sinofool Bochun Bai added a comment -

        A modified patch for trunk version 1043843

        Show
        sinofool Bochun Bai added a comment - A modified patch for trunk version 1043843
        Hide
        sinofool Bochun Bai added a comment -

        Overwrite previous patch which is not includes shims new files.
        This patch is working but needs more works about comments and test cases.
        It is not ready to commit.

        Show
        sinofool Bochun Bai added a comment - Overwrite previous patch which is not includes shims new files. This patch is working but needs more works about comments and test cases. It is not ready to commit.
        Hide
        krystian Krystian Nowak added a comment -

        More general remark connected with this issue: Do you possibly know if there are any plans to release Hive 0.6.x where x>0 (the same way as with 0.4.1 previously) compatible with Hadoop 0.21.0?

        As Hadoop 0.21.0 has been released on 23 August 2010 and Hive 0.6.0 has been released on 19 October 2010 (almost 2 months later)
        it seems highly probable that users would tend to use (or rather try to use) Hive 0.6.0 with newest stable version rather than with 0.20.2 version released on 26 February 2010.

        That's why there is a question if you know whether there are any plans to release - let's say - Hive 0.6.1 targeted for compatibility with Hadoop 0.21.0 before 0.7.0 is released?
        181 of 248 issues have been resolved as of today: https://issues.apache.org/jira/secure/BrowseProject.jspa?id=12310843&subset=-1 so candidly assuming linear progress (shame on me...) should we expect rather 0.7.0 stable in the end of February / beginning of March or rather 0.6.1 might be worth creating before?

        Show
        krystian Krystian Nowak added a comment - More general remark connected with this issue: Do you possibly know if there are any plans to release Hive 0.6.x where x>0 (the same way as with 0.4.1 previously) compatible with Hadoop 0.21.0? As Hadoop 0.21.0 has been released on 23 August 2010 and Hive 0.6.0 has been released on 19 October 2010 (almost 2 months later) it seems highly probable that users would tend to use (or rather try to use) Hive 0.6.0 with newest stable version rather than with 0.20.2 version released on 26 February 2010. That's why there is a question if you know whether there are any plans to release - let's say - Hive 0.6.1 targeted for compatibility with Hadoop 0.21.0 before 0.7.0 is released? 181 of 248 issues have been resolved as of today: https://issues.apache.org/jira/secure/BrowseProject.jspa?id=12310843&subset=-1 so candidly assuming linear progress (shame on me...) should we expect rather 0.7.0 stable in the end of February / beginning of March or rather 0.6.1 might be worth creating before?
        Hide
        namit Namit Jain added a comment -

        Right now, there are no plans to release Hive 0.6.x. (where x > 0) compatible with hadoop 0.21.0

        Do you guys think it is a good time to start wrapping up Hive 0.7 ?
        If we can get 0.7 out by March (with support for Hadoop 0.21), do we still need Hive 0.6.1 ?

        Show
        namit Namit Jain added a comment - Right now, there are no plans to release Hive 0.6.x. (where x > 0) compatible with hadoop 0.21.0 Do you guys think it is a good time to start wrapping up Hive 0.7 ? If we can get 0.7 out by March (with support for Hadoop 0.21), do we still need Hive 0.6.1 ?
        Hide
        krystian Krystian Nowak added a comment -

        Would that 0.7 be much different than what now Amazon is referring to as 0.7 in their EMR: http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?UsingEMR_Hive.html ?
        It seems to confuse people currently as in HIVE-1878's comments:

        Amazon referred to their binaries as 0.7, even though it hadn't been released yet


        (myself as well, that's why I was asking if 0.7 - which I initially thought would be the one Amazon is unofficially using - will be just officially considered as 0.7.0 final/stable/ANY_OTHER_SERIOUS_ADJECTIVE)

        Show
        krystian Krystian Nowak added a comment - Would that 0.7 be much different than what now Amazon is referring to as 0.7 in their EMR: http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?UsingEMR_Hive.html ? It seems to confuse people currently as in HIVE-1878 's comments: Amazon referred to their binaries as 0.7, even though it hadn't been released yet (myself as well, that's why I was asking if 0.7 - which I initially thought would be the one Amazon is unofficially using - will be just officially considered as 0.7.0 final/stable/ANY_OTHER_SERIOUS_ADJECTIVE)
        Hide
        garo Juho Mäkinen added a comment -

        Is this still valid now when 0.7.0 has been released?

        Show
        garo Juho Mäkinen added a comment - Is this still valid now when 0.7.0 has been released?
        Hide
        sinofool Bochun Bai added a comment -

        Juho, trunk(v1089737) still not compiled against hadoop-core-0.21.0
        And there is no notes for 0.21 support in the RELEASE-NOTE

        Show
        sinofool Bochun Bai added a comment - Juho, trunk(v1089737) still not compiled against hadoop-core-0.21.0 And there is no notes for 0.21 support in the RELEASE-NOTE
        Hide
        sinofool Bochun Bai added a comment -

        There are lots of dependencies incompatible with hadoop-0.21.0:
        HadoopShims uses org.apache.hadoop.mapred but 0.21 introducess org.apache.hadoop.mapreduce
        HBase don't support 0.21.0 too.

        Because 0.21.0 offers a great new feature which is split on bz2 files, I have to make hive run ugly.
        After removing HBase functions and hard-coded some 0.21 namespace, simple queries worked.

        A patch is submited. I hope it will useful for someone.
        And a binary can be found here:
        http://sinofool.net/blog/archives/94

        Show
        sinofool Bochun Bai added a comment - There are lots of dependencies incompatible with hadoop-0.21.0: HadoopShims uses org.apache.hadoop.mapred but 0.21 introducess org.apache.hadoop.mapreduce HBase don't support 0.21.0 too. Because 0.21.0 offers a great new feature which is split on bz2 files, I have to make hive run ugly. After removing HBase functions and hard-coded some 0.21 namespace, simple queries worked. A patch is submited. I hope it will useful for someone. And a binary can be found here: http://sinofool.net/blog/archives/94

          People

          • Assignee:
            Unassigned
            Reporter:
            ajpahl1008 AJ Pahl
          • Votes:
            8 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:

              Development