Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-27078

Read Hive materialized view throw MatchError

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0
    • Fix Version/s: 2.4.1, 3.0.0
    • Component/s: SQL
    • Labels:
      None

      Description

      How to reproduce:

      Hive side:

      CREATE TABLE materialized_view_tbl (key INT);
      CREATE MATERIALIZED VIEW view_1 AS SELECT * FROM materialized_view_tbl;  -- Hive 3.x
      CREATE MATERIALIZED VIEW view_1 DISABLE REWRITE AS SELECT * FROM materialized_view_tbl;  -- Hive 2.3.x
      

      Spark side(read from Hive 2.3.x):

      bin/spark-sql --conf spark.sql.hive.metastore.version=2.3.4 --conf spark.sql.hive.metastore.jars=maven
      
      spark-sql> select * from view_1;
      19/03/06 16:33:44 ERROR SparkSQLDriver: Failed in [select * from view_1]
      scala.MatchError: MATERIALIZED_VIEW (of class org.apache.hadoop.hive.metastore.TableType)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTableOption$3(HiveClientImpl.scala:434)
      	at scala.Option.map(Option.scala:163)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTableOption$1(HiveClientImpl.scala:370)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:277)
      

      Spark side(read from Hive 3.1.x):

      bin/spark-sql --conf spark.sql.hive.metastore.version=3.1.1 --conf spark.sql.hive.metastore.jars=maven
      
      spark-sql> select * from view_1;
      19/03/05 19:55:37 ERROR SparkSQLDriver: Failed in [select * from view_1]
      java.lang.NoSuchFieldError: INDEX_TABLE
      	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTableOption$3(HiveClientImpl.scala:438)
      	at scala.Option.map(Option.scala:163)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$getTableOption$1(HiveClientImpl.scala:370)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.$anonfun$withHiveState$1(HiveClientImpl.scala:277)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.liftedTree1$1(HiveClientImpl.scala:215)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.retryLocked(HiveClientImpl.scala:214)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.withHiveState(HiveClientImpl.scala:260)
      	at org.apache.spark.sql.hive.client.HiveClientImpl.getTableOption(HiveClientImpl.scala:368)
      

        Attachments

          Activity

            People

            • Assignee:
              yumwang Yuming Wang
              Reporter:
              yumwang Yuming Wang
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: