Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6774

Implement Parquet complex types backwards-compatiblity rules

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 1.0.0, 1.1.1, 1.2.1, 1.3.0
    • None
    • SQL
    • None

    Description

      Parquet format PR #17 standardized representation of Parquet complex types and listed backwards-compatibility rules. Spark SQL should implement these compatibility rules to improve interoperatability.

      Before, Spark SQL is only compatible with parquet-avro, parquet-hive, and Impala. And it's done in an error prone ad-hoc way, because Parquet format spec didn't explicitly specify complex type structures at the time Spark SQL Parquet support was firstly authored. After fixing this issue, we are expected to be compatible with most (if not all) systems that generated Parquet data in a systematic way by conforming to Parquet format spec and implementing all backwards-compatibility rules.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            lian cheng Cheng Lian
            Cheng Lian Cheng Lian
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment