Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
  3. IMPALA-5113

Buffer pool unpinned invariant does not take into account multiply-pinned bytes

        Details

        • Type: Sub-task
        • Status: Resolved
        • Priority: Major
        • Resolution: Fixed
        • Affects Version/s: Impala 2.9.0
        • Fix Version/s: Impala 2.9.0
        • Component/s: Backend
        • Labels:
          None
        • Epic Color:
          ghx-label-6

          Description

          While working on converting the BufferedBlockMgr test I discovered that the "dirty unpinned bytes <= unused reservation" invariant is too strict. It should be "dirty unpinned bytes <= unused reservation + multiply pinned bytes", since pinning a page multiple times does not require additional buffers.

          I was able to prove the bug by adding a DCHECK for the previous invariant. Pinning a page multiple times could violate the invariant (harmlessly, aside from waiting for some extra I/O).

            Activity

            Hide
            Permalink
            tarmstrong Tim Armstrong added a comment -

            IMPALA-5113: fix dirty unpinned invariant

            There were two bugs:

            • The invariant was too strict and didn't take into account multiple
              pins of pages (which don't use buffers and therefore shouldn't count).
            • The invariant wasn't enforced when reclaiming a clean page.

            Change the logic so that it's implemented in terms of pages/buffers in
            various states (this avoids the reservation double-counting and more
            directly expresses the intent). To aid in this, refactor the page lists
            to use a wrapper that tracks the # of bytes of pages in each list.

            Testing:
            Added a unit test that reproduces the issue and added stricter DCHECKs
            to detect the issue in future.

            Change-Id: I07e08acb6cf6839bfccbd09258c093b1c8252b25
            Reviewed-on: http://gerrit.cloudera.org:8080/6469
            Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
            Tested-by: Impala Public Jenkins
            —

            Show
            tarmstrong Tim Armstrong added a comment - IMPALA-5113 : fix dirty unpinned invariant There were two bugs: The invariant was too strict and didn't take into account multiple pins of pages (which don't use buffers and therefore shouldn't count). The invariant wasn't enforced when reclaiming a clean page. Change the logic so that it's implemented in terms of pages/buffers in various states (this avoids the reservation double-counting and more directly expresses the intent). To aid in this, refactor the page lists to use a wrapper that tracks the # of bytes of pages in each list. Testing: Added a unit test that reproduces the issue and added stricter DCHECKs to detect the issue in future. Change-Id: I07e08acb6cf6839bfccbd09258c093b1c8252b25 Reviewed-on: http://gerrit.cloudera.org:8080/6469 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins —

              People

              • Assignee:
                tarmstrong Tim Armstrong
                Reporter:
                tarmstrong Tim Armstrong
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Development