Any one working on this issue ?

No one has reached out to me for further info or clarification.

Sent via PocketJIRA - www.pocketjira.com

Their are issues open to get hive working with hadoop security. Since security is only in the 0.21 branch, thus someone is working on this.

Currently the people working on security are doing so against features backported to 0.20S (see HIVE-1264 shims for example). This is distinct from 0.21, which only has a partial implementation of security features.

Hi,
I have managed to build Hive (trunk -r 1000628) against Hadoop 0.21.0.

Following is what I had to do to make it work.
1. build Hive as instructed in Hive wiki (ant pacakge).
2. Copy hadoop-0.21.0.tar.gz to $

/build/hadoopcore, untar it, and create/touch the "hadoop-0.21.0.installed" file
3. Apply the patch
4. build Hive again by issuing ant package "-Dhadoop.version=0.21.0 -Doffline=true"

I understand that the patch and process to build is not in the pristine forms, but I would like to get some feedback and see if I can pursue this further.

Thanks.

Hi,

Can you execute the hive-ql in hive client? i got this error below:

$ ./hive
Exception in thread "main" java.lang.RuntimeException: Could not load shims
in class null
at org.apache.hadoop
.hive.shims.ShimLoader.loadShims(ShimLoader.java:86)
at org.apache.hadoop
.hive.shims.ShimLoader.getHadoopShims(ShimLoader.java:62)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:234)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:192)
Caused by: java.lang.NullPointerException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:169)
at org.apache.hadoop
.hive.shims.ShimLoader.loadShims(ShimLoader.java:83)
... 7 more

and cannot execute hive, is it HIVE dont support hadoop0.21.0 now?

Thanks & Regards,
Singo

As per Daisuke's instructions I was able to complile trunk-1000628 against Hadoop 0.21.0.
Unfortunately, execution isn't exactly like Singo's but is still contains a few exceptions:

-bash-3.2$ ./hive
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:185)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 3 more

Also starting hiveserver...

Starting Hive Thrift Server
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/thrift/protocol/TProtocolFactory
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:185)
Caused by: java.lang.ClassNotFoundException: org.apache.thrift.protocol.TProtocolFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
at java.lang.ClassLoader.loadClass(ClassLoader.java:252)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:320)
... 3 more

I've had success with compiling previous releases of 0.6.0 and 0.7.0 (prior to 0.21.0) so I'm
pretty sure my environment is correct for testing this. (ie: Classpath etc)

Daisuke, keep up the good work. I think this JIRA will be renamed DAISUKE-1612 if you can get it worked out.

Hi AJ,

I would like to get more information about your environment because I can successfully start the hive cli session as well as hive thrift server, and they seem to work fine.

Your error messages suggest to me that your class path is not set correctly for some reason.
However those two classes that are reported (org/apache/hadoop/hive/conf/HiveConf and org/apache/thrift/protocol/TProtocolFactory) are present in the $HIVE_HOME/lib folder, namely hive-common-0.7.0.jar and libthrift.jar. Delving further into how the classpath is set, it seems like bin/hive takes care of it.
So I am not sure why your instance fails to locate those classes.

Please let me know if you have any suggestions.

A modified patch for trunk version 1043843

Overwrite previous patch which is not includes shims new files.
This patch is working but needs more works about comments and test cases.
It is not ready to commit.

More general remark connected with this issue: Do you possibly know if there are any plans to release Hive 0.6.x where x>0 (the same way as with 0.4.1 previously) compatible with Hadoop 0.21.0?

As Hadoop 0.21.0 has been released on 23 August 2010 and Hive 0.6.0 has been released on 19 October 2010 (almost 2 months later)
it seems highly probable that users would tend to use (or rather try to use) Hive 0.6.0 with newest stable version rather than with 0.20.2 version released on 26 February 2010.

That's why there is a question if you know whether there are any plans to release - let's say - Hive 0.6.1 targeted for compatibility with Hadoop 0.21.0 before 0.7.0 is released?
181 of 248 issues have been resolved as of today: https://issues.apache.org/jira/secure/BrowseProject.jspa?id=12310843&subset=-1 so candidly assuming linear progress (shame on me...) should we expect rather 0.7.0 stable in the end of February / beginning of March or rather 0.6.1 might be worth creating before?

Right now, there are no plans to release Hive 0.6.x. (where x > 0) compatible with hadoop 0.21.0

Do you guys think it is a good time to start wrapping up Hive 0.7 ?
If we can get 0.7 out by March (with support for Hadoop 0.21), do we still need Hive 0.6.1 ?

Would that 0.7 be much different than what now Amazon is referring to as 0.7 in their EMR: http://docs.amazonwebservices.com/ElasticMapReduce/latest/DeveloperGuide/index.html?UsingEMR_Hive.html ?
It seems to confuse people currently as in HIVE-1878's comments:

Amazon referred to their binaries as 0.7, even though it hadn't been released yet

Is this still valid now when 0.7.0 has been released?

Juho, trunk(v1089737) still not compiled against hadoop-core-0.21.0
And there is no notes for 0.21 support in the RELEASE-NOTE

There are lots of dependencies incompatible with hadoop-0.21.0:
HadoopShims uses org.apache.hadoop.mapred but 0.21 introducess org.apache.hadoop.mapreduce
HBase don't support 0.21.0 too.

Because 0.21.0 offers a great new feature which is split on bz2 files, I have to make hive run ugly.
After removing HBase functions and hard-coded some 0.21 namespace, simple queries worked.

A patch is submited. I hope it will useful for someone.
And a binary can be found here:
http://sinofool.net/blog/archives/94