Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6471

Metastore schema should only be a subset of parquet schema to support dropping of columns using replace columns

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 1.3.0
    • Fix Version/s: 1.3.1, 1.4.0
    • Component/s: SQL
    • Labels:
      None

      Description

      Currently in the parquet relation 2 implementation, error is thrown in case merged schema is not exactly the same as metastore schema.
      But to support cases like deletion of column using replace column command, we can relax the restriction so that even if metastore schema is a subset of merged parquet schema, the query will work.

        Attachments

        Issue Links

          Activity

            People

            • Assignee:
              saucam Yash Datta
              Reporter:
              saucam Yash Datta

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment