Uploaded image for project: 'IMPALA'
  1. IMPALA
  2. IMPALA-3200 Replace BufferedBlockMgr with new buffer pool
  3. IMPALA-5113

Buffer pool unpinned invariant does not take into account multiply-pinned bytes

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None
    • Epic Color:
      ghx-label-6

      Description

      While working on converting the BufferedBlockMgr test I discovered that the "dirty unpinned bytes <= unused reservation" invariant is too strict. It should be "dirty unpinned bytes <= unused reservation + multiply pinned bytes", since pinning a page multiple times does not require additional buffers.

      I was able to prove the bug by adding a DCHECK for the previous invariant. Pinning a page multiple times could violate the invariant (harmlessly, aside from waiting for some extra I/O).

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5113: fix dirty unpinned invariant

        There were two bugs:

        • The invariant was too strict and didn't take into account multiple
          pins of pages (which don't use buffers and therefore shouldn't count).
        • The invariant wasn't enforced when reclaiming a clean page.

        Change the logic so that it's implemented in terms of pages/buffers in
        various states (this avoids the reservation double-counting and more
        directly expresses the intent). To aid in this, refactor the page lists
        to use a wrapper that tracks the # of bytes of pages in each list.

        Testing:
        Added a unit test that reproduces the issue and added stricter DCHECKs
        to detect the issue in future.

        Change-Id: I07e08acb6cf6839bfccbd09258c093b1c8252b25
        Reviewed-on: http://gerrit.cloudera.org:8080/6469
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5113 : fix dirty unpinned invariant There were two bugs: The invariant was too strict and didn't take into account multiple pins of pages (which don't use buffers and therefore shouldn't count). The invariant wasn't enforced when reclaiming a clean page. Change the logic so that it's implemented in terms of pages/buffers in various states (this avoids the reservation double-counting and more directly expresses the intent). To aid in this, refactor the page lists to use a wrapper that tracks the # of bytes of pages in each list. Testing: Added a unit test that reproduces the issue and added stricter DCHECKs to detect the issue in future. Change-Id: I07e08acb6cf6839bfccbd09258c093b1c8252b25 Reviewed-on: http://gerrit.cloudera.org:8080/6469 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins —

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development