[SPARK-26145] Not Able To Read Data From Hive 3.0 Using Spark 2.3

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 2.3.0, 2.3.1
• Component/s: Java API, SQL

Environment

Hive Version:- 3.1.0.3.0.0.0-1634

Hbase Version:- 2.0.0.3.0.0.0-1634

Spark Version:- 2.3.1.3.0.0.0-1634

• Assignee: Unassigned
• Reporter: Avasyu Gupta
• Created: 22/Nov/18 10:11
• Updated: 06/Jun/19 22:43
• Resolved: 26/Nov/18 16:27

Description

Hello Team,

 

We are trying to read data from hive tables using Spark SQL but are unable to do so. Following are the steps we are following to achieve the same:-

1. Created certain tables in Hive 3.1.0 and linked them to the tables of Hbase 2.0.0 using HbaseStorageHandler SerDe.
2. All the configuration related to hive including spark.sql.warehouse.dir, thrift server uri, zookeeper details etc. are being provided using SparkConf.
3. We are then creating the Spark Session as SparkSession spark = SparkSession.builder().config(conf).enableHiveSupport()
   .getOrCreate();
4. Then using SQLContext we are trying to read data from the hive table by:- sqlContext.sql("select * from db_name.table_name").show();

At this step we are facing the error as:- java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found (Full logs attached)

We are including the hive-hbase-handler jar and all the other required jars in our commonLib and specifying the absolute path to our commonLib using the --jars option in our spark-submit, yet we are unable to find a wayout to resolve this error.

We read in the Spark's official documentation that it is still supporting upto Hive 2.1. So is there another way to connect to Hive 3.0 using Spark 2.3?

Issue Links

• duplicates: SPARK-24360 Support Hive 3.1 metastore
• links to: GitHub Pull Request #24688