Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-7340

Use latest parquet release 1.6.0 in spark

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 1.3.1
    • None
    • SQL
    • None

    Description

      Bump the used parquet version from 1.6.0rc3 to 1.6.0.
      This brings in major improvement in the form of reading footers in the task side instead of the driver, removing a major bottleneck

      Attachments

        Issue Links

          duplicates

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-6566 Update Spark to use the latest version of Parquet libraries

          • Major - Major loss of function.
          • Resolved
          is part of

          Bug - A problem which impairs or prevents the functions of the product. SPARK-6795 Avoid reading Parquet footers on driver side when an global arbitrative schema is available

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Pull request #5888 [Github] Pull Request #5888 (saucam)

          Activity

            People

              Unassigned Unassigned
              saucam Yash Datta
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: