Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13446

Spark need to support reading data from Hive 2.0.0 metastore

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 2.2.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Spark provided HIveContext class to read data from hive metastore directly. While it only supports hive 1.2.1 version and older. Since hive 2.0.0 has released, it's better to upgrade to support Hive 2.0.0.

      16/02/23 02:35:02 INFO metastore: Trying to connect to metastore with URI thrift://hsw-node13:9083
      16/02/23 02:35:02 INFO metastore: Opened a connection to metastore, current connections: 1
      16/02/23 02:35:02 INFO metastore: Connected to metastore.
      Exception in thread "main" java.lang.NoSuchFieldError: HIVE_STATS_JDBC_TIMEOUT
              at org.apache.spark.sql.hive.HiveContext.configure(HiveContext.scala:473)
              at org.apache.spark.sql.hive.HiveContext.metadataHive$lzycompute(HiveContext.scala:192)
              at org.apache.spark.sql.hive.HiveContext.metadataHive(HiveContext.scala:185)
              at org.apache.spark.sql.hive.HiveContext$$anon$1.<init>(HiveContext.scala:422)
              at org.apache.spark.sql.hive.HiveContext.catalog$lzycompute(HiveContext.scala:422)
              at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:421)
              at org.apache.spark.sql.hive.HiveContext.catalog(HiveContext.scala:72)
              at org.apache.spark.sql.SQLContext.table(SQLContext.scala:739)
              at org.apache.spark.sql.SQLContext.table(SQLContext.scala:735)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                smilegator Xiao Li
                Reporter:
                Lifeng Wang Lifeng Wang
              • Votes:
                3 Vote for this issue
                Watchers:
                29 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: