[DRILL-481] libthrift-0.8.0 causes drillbit to fail while querying Hive tables

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.4.0

Environment

Centos 6.4 + Maven 3.0.5

• Assignee: Steven Phillips
• Reporter: Abhinav Chawade
• Created: 02/Apr/14 23:51
• Updated: 31/Jul/14 06:33
• Resolved: 20/Apr/14 04:10

Description

When drill is built on any system, libthrift-0.8.0 and libthrift-0.9.0 both get pulled in as part of multiple projects. The attached deps.txt file shows dependency tree. When drillbit is started and a query against Hive table is fired, drillbit crashes with following NoSuchMethodError.
Caused by: java.lang.NoSuchMethodError: org.apache.thrift.EncodingUtils.setBit(BIZ)B
at org.apache.hadoop.hive.metastore.api.StorageDescriptor.setCompressedIsSet(StorageDescriptor.java:442)
at org.apache.hadoop.hive.metastore.api.StorageDescriptor.<init>(StorageDescriptor.java:223)
at org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1019)
at org.apache.hadoop.hive.metastore.ObjectStore.convertToStorageDescriptor(ObjectStore.java:1034)
at org.apache.hadoop.hive.metastore.ObjectStore.convertToTable(ObjectStore.java:889)
at org.apache.hadoop.hive.metastore.ObjectStore.getTable(ObjectStore.java:760)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingRawStore.invoke(RetryingRawStore.java:124)
at com.sun.proxy.$Proxy18.getTable(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table(HiveMetaStore.java:1371)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:103)
at com.sun.proxy.$Proxy19.get_table(Unknown Source)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:854)
at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$TableLoader.load(HiveSchemaFactory.java:144)
at org.apache.drill.exec.store.hive.schema.HiveSchemaFactory$TableLoader.load(HiveSchemaFactory.java:131)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
... 35 more

It turns out the $DRIL_HOME/lib directory contains libthrift-0.8.0.jar which doesn't have setBit method in EncodingUtils.java but it is present in 0.9.0.

https://github.com/apache/thrift/blob/0.8.x/lib/java/src/org/apache/thrift/EncodingUtils.java
https://github.com/apache/thrift/blob/0.9.x/lib/java/src/org/apache/thrift/EncodingUtils.java

Only 0.9.0 should be included in Drill's distribution tarball if possible