Major Description
Kudu starts flushing aggressively when memory usage exceeds 60% of the hard limit. In pathological cases, this can cause Kudu to flush an extremely large amount of small rowsets.
For example, if -block_cache_capacity_mb is set to over 60% of the hard limit, and the block cache fills, then Kudu will always be under memory pressure. Flushing won't ever be able to reduce memory usage under the aggressive flushing threshold. However, Kudu will still flush, producing lots of tiny rowsets. This can eventually cause problems like KUDU-2317.
Attached is the rowset diagram from a tablet showing this phenomenon. To produce this I
1. Ran the tablet servers with -memory_limit_hard_bytes=1048576000 (1GiB) and -block_cache_capacity_mb=750.
2. Started an insert workload (single-threaded so insert would be sequential): build/latest/bin/kudu perf loadgen -keep_auto_table -num_threads=1 -num_rows_per_thread=100000000.
3. Waited a bit for tablet servers to have a gigabyte or two of data.
4. Ran a ksck checksum set to cache blocks: build/latest/bin/kudu cluster ksck -checksum_scan -checksum_cache_blocks.
