Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5463 Improve Parquet support (reliability, performance, and error messages)
  3. SPARK-8125

Accelerate ParquetRelation2 metadata discovery

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 1.4.0
    • 1.5.0
    • SQL
    • None

    Description

      For large Parquet tables (e.g., with thousands of partitions), it can be very slow to discover Parquet metadata for schema merging and generating splits for Spark jobs. We need to accelerate this processes. One possible solution is to do the discovery via a distributed Spark job.

      Attachments

        Issue Links

          is duplicated by

          Improvement - An improvement or enhancement to an existing feature or task. SPARK-9347 spark load of existing parquet files extremely slow if large number of files

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. SPARK-6795 Avoid reading Parquet footers on driver side when an global arbitrative schema is available

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          relates to

          Sub-task - The sub-task of the issue SPARK-9072 Parquet : Writing data to S3 very slowly

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved
          links to

          Pull request #7396 [Github] Pull Request #7396 (liancheng)

          Pull request #7664 [Github] Pull Request #7664 (liancheng)

          Activity

            People

              lian cheng Cheng Lian
              lian cheng Cheng Lian
              Cheng Lian Cheng Lian
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: