Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
  3. IMPALA-5677

Consider ways to reduce the accumulation of clean pages when executing large spilling queries

    Details

      Description

      By design, the new buffer pool aggressively uses available memory to hold onto spilled data in memory.

      On some stress tests runs (and in other testing) I've seen the clean pages memory blow up and cause Impala to use more memory than one might expect based on the query memory limits (but not more than the process limit, so we're not going outside of the configured resource constraints).

      I think this may hurt performance more than it helps in many cases because it will squeeze out the OS buffer cache.

        Attachments

          Activity

            People

            • Assignee:
              tarmstrong Tim Armstrong
              Reporter:
              tarmstrong Tim Armstrong
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: