Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-2352

PAGG::ConstructIntermediateTuple() does not handle correctly small buffers getting full

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.0, Impala 2.1, Impala 2.2, Impala 2.3.0
    • Fix Version/s: Impala 2.3.0
    • Component/s: None

      Description

      The small buffer optimization patch, which includes also a fix for IMPALA-2330, exposed issues with the buffered-tuple-stream, buffered-block-mgr and how PAGG and PHJ handle various cases.

      If a stream is using small buffers and those buffers get full, then the next call to Stream::AllocateRow() by PAGG will fail, because we do not auto-switch to IO-buffers. We would deal with this failure with a hammer: by spilling a partition and switching to IO-buffers all streams. Then, in the next partitioning level we would start all the partitions with IO-buffers (no small buffers) and we would bypass this problem.

      The less harmful but important result of this issue is excessive/pre-mature spilling. That is, we end up spilling partitions of PAGG just because the small buffers one of the streams was ending up full, and even though there is plenty of memory available. The most harmful result is queries failing with OOM because of the additional memory needed for the unneeded repartitions.

        Attachments

          Activity

            People

            • Assignee:
              ippokratis Ippokratis Pandis
              Reporter:
              ippokratis Ippokratis Pandis
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: