[SPARK-13446] Spark need to support reading data from Hive 2.0.0 metastore

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.6.0
• Fix Version/s: 2.2.0
• Component/s: SQL
• Assignee: Xiao Li
• Reporter: Lifeng Wang
• Created: 23/Feb/16 07:14
• Updated: 01/Oct/19 20:17
• Resolved: 04/Mar/17 01:00

Description

Spark provided HIveContext class to read data from hive metastore directly. While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has released, it's better to upgrade to support Hive 2.0.0.

16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI thrift://hsw-node13:9083
16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current connections: 1
16/02/23 02:35:02 INFO metastore: Connected to metastore.
Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
        at org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
        at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
        at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
        at org.apache.spark.sql.hive.HiveContext$$anon$1.<init>(HiveContext.scala:422)
        at org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
        at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:421)
        at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:72)
        at org.apache.spark.sql.SQLContext.table(SQLContext.scala:739)
        at org.apache.spark.sql.SQLContext.table(SQLContext.scala:735)

Issue Links

• is duplicated by: SPARK-18112 Spark2.x does not support read data from Hive 2.x metastore
• is duplicated by: SPARK-19076 Upgrade Hive dependence to Hive 2.x
• is duplicated by: SPARK-19321 Support Hive 2.x's metastore
• is related to: SPARK-19076 Upgrade Hive dependence to Hive 2.x
• links to: [Github] Pull Request #17061 (gatorsmile)