Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-11402

getPartialCatalogObject fails with OOM with huge number of partitions

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Critical
    • Resolution: Unresolved
    • None
    • None
    • Catalog
    • None
    • ghx-label-1

    Description

      The response size of getPartialCatalogObject depends on the number of partitions in the request. Even with the optimization of IMPALA-7501, the response size could still exceeds the 2GB byte array limit if requesting all partitions of a huge table.

      We should add flag to limit the number of partitions in a single getPartialCatalogObject request. When more partitions are required, fetch them in different batches.

      Attachments

        Issue Links

          relates to

          Bug - A problem which impairs or prevents the functions of the product. IMPALA-9896 OutOfMemoryError: Requested array size exceeds VM limit when LocalCatalog is enabled

          • Critical - Crashes, loss of data, severe memory leak.
          • Resolved

          Activity

            People

              stigahuang Quanlong Huang
              stigahuang Quanlong Huang
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated: