[AURORA-74] Write snapshots and backups in a way that requires less memory

• Type: Story
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Component/s: Scheduler
• Assignee: Unassigned
• Reporter: Bill Farner
• Created: 22/Jan/14 03:38
• Updated: 17/Apr/15 22:27

Description

Snapshots and backups currently involve passing around instances of the Snapshot struct. These are fully-hydrated copies of the entire storage. As a result, a scheduler process needs a constant multiplier of heap space headroom in order to perform these operations without encountering OutOfMemoryError s. Fixing this will require the ability to 'stream' fixed-size parts of storage from MemStorage to the backing store (currently an org.apache.mesos.Log), or a file in the case of backups. The reverse is also necessary — a snapshot must be read from a sequence of {{org.apache.mesos.Log.Entry}}s and fed into the in-memory storage.

Issue Links

• is related to: AURORA-722 snapshot performance issues
• is related to: AURORA-930 GC performance issues with snapshot deduplication on large clusters