[MESOS-6267] Mesos master should periodically compact the replicated log

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: replicated log
• Assignee: Unassigned
• Reporter: Daniel Robinson
• Created: 28/Sep/16 18:28
• Updated: 03/Oct/16 16:20

Description

The mesos master does not periodically compact the replicated log's leveldb, it grows without bound. Compaction only occurs on startup, therefore in large clusters (10k+ agents) where the master is restarted infrequently compaction at startup can take 2+ minutes to complete. At Twitter we've seen the DB grow to 170+ GB in less than a month and compaction take > 2 minutes. Compaction reduces storage usage from 160 GB to roughly 50 MB.

I0928 06:06:20.776908 19725 leveldb.cpp:174] Opened db in 763.082271ms
I0928 06:08:41.636145 19725 leveldb.cpp:181] Compacted db in 2.34765004808333mins

The replicated log should not grow without bounds.

Issue Links

• is related to: MESOS-184 Log has a space leak
• is related to: MESOS-1944 Support streaming read for replicated log
• is related to: MESOS-3180 Replicated log (registry) may bloat under some conditions