[BOOKKEEPER-181] Scale hedwig

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Component/s: bookkeeper-server, hedwig-server
• Assignee: Sijie Guo
• Reporter: Sijie Guo
• Created: 07/Mar/12 01:27
• Updated: 14/Jan/13 14:21
• Resolved: 18/Dec/12 05:56

Description

Current implementation of Hedwig and BookKeeper is designed to scale to hundreds of thousands of topics, but now we are looking at scaling them to tens to hundreds of millions of topics, using a scalable key/value store such as HBase.

Issue Links

• is related to: BOOKKEEPER-249 Revisit garbage collection algorithm in Bookie server
• is related to: BOOKKEEPER-268 Implement Metadata plugin APIs for ZooKeeper

Sub-Tasks

• improve ledger manager interface to remove zookeeper dependency on metadata operations.: Closed
• Provide a MetaStore interface, and a mock implementation.: Closed
• implement a MetaStore based ledger manager for bookkeeper client.: Closed
• implement a MetaStore based active ledger manager for bookie server.: Resolved
• Need a ledger manager like interface to manage metadata operations in Hedwig: Closed
• Create a topic manager using versioned write for leader election: Closed
• Implement a meta store based hedwig metadata manager.: Closed
• Improve Hedwig Console to use Hedwig Metadata Manager.: Closed
• add documentation for bookkeeper ledger manager interface.: Closed
• add documentation for hedwig metadata manager interface.: Closed
• Make Write/Delete SubscriptionData Restricted to Version: Closed
• Rename metastore mock implementation to InMemory implementation: Closed
• add documentation for MetaStore interface: Closed