Uploaded image for project: 'Kudu'
  1. Kudu
  2. KUDU-2318

Kudu flushes aggressively even if it can't relieve memory pressure.

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.6.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Kudu starts flushing aggressively when memory usage exceeds 60% of the hard limit. In pathological cases, this can cause Kudu to flush an extremely large amount of small rowsets.

      For example, if -block_cache_capacity_mb is set to over 60% of the hard limit, and the block cache fills, then Kudu will always be under memory pressure. Flushing won't ever be able to reduce memory usage under the aggressive flushing threshold. However, Kudu will still flush, producing lots of tiny rowsets. This can eventually cause problems like KUDU-2317.

      Attached is the rowset diagram from a tablet showing this phenomenon. To produce this I
      1. Ran the tablet servers with -memory_limit_hard_bytes=1048576000 (1GiB) and -block_cache_capacity_mb=750.
      2. Started an insert workload (single-threaded so insert would be sequential): build/latest/bin/kudu perf loadgen -keep_auto_table -num_threads=1 -num_rows_per_thread=100000000.
      3. Waited a bit for tablet servers to have a gigabyte or two of data.
      4. Ran a ksck checksum set to cache blocks: build/latest/bin/kudu cluster ksck -checksum_scan -checksum_cache_blocks.

        Attachments

        1. rowsetblackbarofdeath.png
          152 kB
          William Berkeley

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                wdberkeley William Berkeley
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: