Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5463 Improve Parquet support (reliability, performance, and error messages)
  3. SPARK-7743

Upgrade Parquet to 1.7

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.5.0
    • SQL
    • None

    Description

      There are many outstanding issues with the parquet format that have been resolved between the version depended on by spark (1.6.0rc3 as of spark 1.3.1) and the most recent parquet release (1.6.0).

      Some of these are things include not supporting schema migration when using parquet with avro, not supporting summary metadata in the parquet footers causing null pointer exceptions reading, and many others.

      See https://github.com/apache/parquet-mr/blob/master/CHANGES.md#version-160 for the full list of fixes.

      Attachments

        Issue Links

          is related to

          Sub-task - The sub-task of the issue SPARK-4258 NPE with new Parquet Filters

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-5451 And predicates are not properly pushed down

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Sub-task - The sub-task of the issue SPARK-2551 Cleanup FilteringParquetRowInputFormat

          • Minor - Minor loss of function, or other problem where easy workaround is present.
          • Resolved
          links to

          Web Link Github Pull Request

          Activity

            People

              eggsby Thomas Omans
              eggsby Thomas Omans
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: