Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-6774 Implement Parquet complex types backwards-compatiblity rules
  3. SPARK-8848

Write Parquet LISTs and MAPs conforming to Parquet format spec

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 1.0.2, 1.1.1, 1.2.2, 1.3.1, 1.4.0
    • 1.6.0
    • SQL
    • None

    Description

      Parquet format PR #17 standardized structures of Parquet complex types (LIST & MAP). Spark SQL should follow this spec and write Parquet data conforming to the standard.

      Note that although currently Parquet files written by Spark SQL is non-standard (because Parquet format spec wasn't clear about this part when Spark SQL Parquet support was authored), it's still compatible with the most recent Parquet format spec, because the format we use is covered by the backwards-compatibility rules.

      Attachments

        Activity

          People

            lian cheng Cheng Lian
            lian cheng Cheng Lian
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: