Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-1755

Improve tablet disk space estimation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1.0
    • Fix Version/s: 1.6.0
    • Component/s: supportability, tablet
    • Labels:
      None
    • Target Version/s:

      Description

      (Prompted by this user post)

      The on-disk size of tablets as reported by the Kudu web UI omits some minor as well as some major sources of space consumption. I'm listing them all here for posterity.

      1. Bloom file and composite index file usage. According to this gerrit (warning: internal link), it's because we also use the rowset estimate to determine how much IO will be generated were we to compact that rowset, and bloom/composite index files aren't touched in compaction.
      2. UNDO file usage. This seems like a more glaring omission, especially for mutation-heavy workloads like the one reported in the mailing list. But, the current REDO-only estimate factors into major delta compaction decision making by the maintenance manager, so maybe there's a good reason there too.
      3. Log block manager block size rounding. The LBM rounds up Kudu blocks to the nearest filesystem block size to improve hole punching space reclamation. A side effect is that some space is lost to external fragmentation.
      4. Log block manager metadata overhead. Every container has a .metadata file, and we don't factor that into space utilization.
      5. Other files, such as the tablet superblock, WAL segments, and cmeta.

      I expect the first two items to be the largest, so we should work on addressing them. Lets decouple the UI-based estimate from the MM path so our reporting can be more accurate while still allowing the MM to make good decisions.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                wdberkeley William Berkeley
                Reporter:
                adar Adar Dembo
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: