Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5463 Improve Parquet support (reliability, performance, and error messages)
  3. SPARK-8580

Test Parquet interoperability and compatibility with other libraries/systems

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.5.0
    • Fix Version/s: 1.5.0
    • Component/s: SQL
    • Labels:
      None

      Description

      As we are implementing Parquet backwards-compatibility rules for Spark 1.5.0 to improve interoperability with other systems (reading non-standard Parquet files they generate, and generating standard Parquet files), it would be good to have a set of standard test Parquet files generated by various systems/tools (parquet-thrift, parquet-avro, parquet-hive, Impala, and old versions of Spark SQL) to ensure compatibility.

        Attachments

          Activity

            People

            • Assignee:
              lian cheng Cheng Lian
              Reporter:
              lian cheng Cheng Lian
            • Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: