Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-8942

Set file format specific values for split sizes on non-block stores

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • Impala 3.4.0
    • Frontend
    • None
    • ghx-label-12

    Description

      Parquet scans on non-block based storage systems (e.g. S3, ADLS, etc.) can suffer from uneven scan range assignment due to the behavior described in IMPALA-3453. The frontend should set different split sizes depending on the file type and file system.

      Attachments

        Issue Links

          is blocked by

          Test - A new unit, integration or system test. IMPALA-8944 Update and re-enable S3PlannerTest

          • Major - Major loss of function.
          • Resolved
          is related to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-3453 S3 : Uneven split sizes are generated for Parquet causing execution skew

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              stakiar Sahil Takiar
              stakiar Sahil Takiar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: