Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: Impala 2.9.0
    • Fix Version/s: Impala 2.9.0
    • Component/s: Backend
    • Labels:
      None

      Description

      We should clean-up the profile counters that are added to each node so that irrelevant info is hidden and relevant info is easy to find:

      • Hide the limit if it is MAX_INT64
      • Remove the BufferPool prefix and move into a "Buffer pool" section, or similar
      • Remove or rename potentially confusing counters, e.g.
        • the cumulative allocation counters should be removed or renamed to be clearer that they are cumulative
        • PeakUnpinnedBytes is confusing unless you understand the internals

        Activity

        Hide
        tarmstrong Tim Armstrong added a comment -

        IMPALA-5166: clean up BufferPool counters

        Misc changes to improve usability of the profiles.

        • Separate out detailed BufferPool metrics into a "Buffer pool"
          sub-profile.
        • Only create the limit counter if there is a limit
        • Show BufferPool using in query MemTracker (it was accidentally
          disabled before because there was no query-level profile).
        • Reduce clutter in MemTracker dump by only showing buffer pool
          reservation, not usage (the usage was misleading anyway because
          it didn't include child usage).
        • Remove TotalUnpinnedBytes, which had limited value - WriteIoBytes
          and PeakUnpinnedBytes can answer most of the same questions - i.e.
          did it unpin any pages, and how many did it need to write to disk.
        • Add buffer pool metrics to /memz (if buffer pool is enabled) and
          reorder /memz so more useful information is up the top.

        Change-Id: I34b7f4d94c3d396ac89026c7559d6b2c6e02697c
        Reviewed-on: http://gerrit.cloudera.org:8080/6690
        Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com>
        Tested-by: Impala Public Jenkins

        Show
        tarmstrong Tim Armstrong added a comment - IMPALA-5166 : clean up BufferPool counters Misc changes to improve usability of the profiles. Separate out detailed BufferPool metrics into a "Buffer pool" sub-profile. Only create the limit counter if there is a limit Show BufferPool using in query MemTracker (it was accidentally disabled before because there was no query-level profile). Reduce clutter in MemTracker dump by only showing buffer pool reservation, not usage (the usage was misleading anyway because it didn't include child usage). Remove TotalUnpinnedBytes, which had limited value - WriteIoBytes and PeakUnpinnedBytes can answer most of the same questions - i.e. did it unpin any pages, and how many did it need to write to disk. Add buffer pool metrics to /memz (if buffer pool is enabled) and reorder /memz so more useful information is up the top. Change-Id: I34b7f4d94c3d396ac89026c7559d6b2c6e02697c Reviewed-on: http://gerrit.cloudera.org:8080/6690 Reviewed-by: Tim Armstrong <tarmstrong@cloudera.com> Tested-by: Impala Public Jenkins —

          People

          • Assignee:
            tarmstrong Tim Armstrong
            Reporter:
            tarmstrong Tim Armstrong
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development