[KUDU-1830] Reduce Kudu WAL log disk usage

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: consensus, log
• Labels: data-scalability
• Target Version/s: 1.4.0
• Assignee: Todd Lipcon
• Reporter: Juan Yu
• Created: 11/Jan/17 01:38
• Updated: 31/May/17 21:41
• Resolved: 31/May/17 21:41

Description

WAL log can take significent disk space. So far there are some config to limit it. but it can go very high.
WAL log size = #tablets * log_segment_size_mb * log segments (1 if there is write ops to this tablet, can go up to log_max_segments_to_retain)
Logs are retained even if there is no write for a while.

We could reduce the WAL log usage by

• reduce min_segments_to_retain to 1 instead of 2, a
• reduce steady state consumption of idle tablets, roll a WAL if it has had no writes for a few minutes and size more than a MB or two so that "idle" tablets have 0 WAL space consumed

Issue Links

• relates to: KUDU-1755 Improve tablet disk space estimation