Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-35313

Hive MetaException attempting to get partition metadata by filter from Hive

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.0.1
    • Fix Version/s: None
    • Component/s: Spark Submit, SQL
    • Labels:
      None
    • Environment:

      Databricks runtime version 7.5 (includes Apache Spark 3.0.1, Scala 2.12)

      Description

      Got an error while running a code through Airflow DAG.

      Exception while running an ETL job on an External table created on Hive stored as parquet in S3 with AWS Glue as metastore. Here's the error message:

       

      java.lang.RuntimeException: Caught Hive MetaException attempting to get partition metadata by filter from Hive. You can set the Spark configuration setting spark.sql.hive.manageFilesourcePartitions to false to work around this problem, however this will result in degraded performance. Please report a bug: https://issues.apache.org/jira/browse/SPARK |

       

      Caused by: MetaException(message:Unknown exception occurred. (Service: AWSGlue; Status Code: 500; Error Code: InternalServiceException; Request ID: 73267997-1795-45a3-965f-8bb2a6b7b3ac))

       

      Exact issue occurred while running on Databricks notebook as well. Screenshot attached for both cases.

        Attachments

        1. spark_issue.JPG
          36 kB
          Kaushik Muniandi
        2. spark_issue_databricks.JPG
          46 kB
          Kaushik Muniandi

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mkk1490 Kaushik Muniandi
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: