Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-5463 Improve Parquet support (reliability, performance, and error messages)
  3. SPARK-2551

Cleanup FilteringParquetRowInputFormat

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 1.0.1, 1.0.2
    • None
    • SQL
    • None

    Description

      To workaround PARQUET-16 and fix SPARK-2119, we did some reflection hacks in FilteringParquetRowInputFormat. This should be cleaned up once PARQUET-16 is fixed.

      A PR for PARQUET-16 is here.

      Attachments

        Issue Links

          relates to

          Sub-task - The sub-task of the issue SPARK-7743 Upgrade Parquet to 1.7

          • Major - Major loss of function.
          • Resolved

          Bug - A problem which impairs or prevents the functions of the product. PARQUET-16 Unnecessary getFileStatus() calls on all part-files in ParquetInputFormat.getSplits

          • Major - Major loss of function.
          • Open

          Bug - A problem which impairs or prevents the functions of the product. SPARK-2119 Reading Parquet InputSplits dominates query execution time when reading off S3

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Task - A task that needs to be done. SPARK-9095 Removes old Parquet support code

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              lian cheng Cheng Lian
              lian cheng Cheng Lian
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: