Uploaded image for project: 'Mesos'
  1. Mesos
  2. MESOS-6267

Mesos master should periodically compact the replicated log

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: replicated log
    • Labels:
      None

      Description

      The mesos master does not periodically compact the replicated log's leveldb, it grows without bound. Compaction only occurs on startup, therefore in large clusters (10k+ agents) where the master is restarted infrequently compaction at startup can take 2+ minutes to complete. At Twitter we've seen the DB grow to 170+ GB in less than a month and compaction take > 2 minutes. Compaction reduces storage usage from 160 GB to roughly 50 MB.

      I0928 06:06:20.776908 19725 leveldb.cpp:174] Opened db in 763.082271ms
      I0928 06:08:41.636145 19725 leveldb.cpp:181] Compacted db in 2.34765004808333mins
      

      The replicated log should not grow without bounds.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                drobinson Daniel Robinson
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: