Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-26145

Not Able To Read Data From Hive 3.0 Using Spark 2.3

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.3.0, 2.3.1
    • Fix Version/s: None
    • Component/s: Java API, SQL
    • Labels:
      None
    • Environment:

      Hive Version:- 3.1.0.3.0.0.0-1634

      Hbase Version:- 2.0.0.3.0.0.0-1634

      Spark Version:- 2.3.1.3.0.0.0-1634

      Description

      Hello Team,

       

      We are trying to read data from hive tables using Spark SQL but are unable to do so. Following are the steps we are following to achieve the same:-

      1. Created certain tables in Hive 3.1.0 and linked them to the tables of Hbase 2.0.0 using HbaseStorageHandler SerDe.
      2. All the configuration related to hive including spark.sql.warehouse.dir, thrift server uri, zookeeper details etc. are being provided using SparkConf.
      3. We are then creating the Spark Session as SparkSession spark = SparkSession.builder().config(conf).enableHiveSupport()
        .getOrCreate();
      4. Then using SQLContext we are trying to read data from the hive table by:- sqlContext.sql("select * from db_name.table_name").show();

      At this step we are facing the error as:- java.lang.ClassNotFoundException Class org.apache.hadoop.hive.hbase.HBaseSerDe not found (Full logs attached)

      We are including the hive-hbase-handler jar and all the other required jars in our commonLib and specifying the absolute path to our commonLib using the --jars option in our spark-submit, yet we are unable to find a wayout to resolve this error.

      We read in the Spark's official documentation that it is still supporting upto Hive 2.1. So is there another way to connect to Hive 3.0 using Spark 2.3?

        Attachments

        1. Logs.txt
          16 kB
          Avasyu Gupta

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Avasyu Gupta Avasyu Gupta
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: