[KUDU-2643] Add tools to do basic analysis on the metrics logs

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.8.0
• Labels: hackathon roadmap-candidate supportability
• Assignee: William Berkeley
• Reporter: William Berkeley
• Created: 17/Dec/18 21:16
• Updated: 02/Nov/22 12:12

Description

KUDU-2597 tracks adding a tool to parse the metrics logs. We should also add (probably as Python scripts) some tools for analyzing the metrics logs:

• Finding tablets with unusual performance characteristics: longest apply|prepare|replicate|write|updateconsensus times
• Finding servers with the most disk activity
• Finding servers with slows scanners
• Finding replicas that are largest on disk or in-memory
• Characterizing workloads of tables and tablets (insert/upserts/deletes/updates/duplicate key inserts/pk lookups + times/op)
• Compaction (average height, delta size, compaction times)
• Log performance (append latency, sync latency, throughput)

Some of this can be rules-based, i.e. if metric X is > constant A) and some should be pattern-based (most tablets' histograms look different than this tablet's).

Issue Links

• is related to: KUDU-3060 Add a tool to identify potential performance bottlenecks
• relates to: KUDU-2597 Add CLI tool to parse metrics from diagnostics log