[KAFKA-50] kafka intra-cluster replication support

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Assignee: Jun Rao
• Reporter: Jun Rao
• Created: 19/Jul/11 21:32
• Updated: 13/Jun/13 15:34
• Resolved: 13/Jun/13 15:34

Description

Currently, Kafka doesn't have replication. Each log segment is stored in a single broker. This limits both the availability and the durability of Kafka. If a broker goes down, all log segments stored on that broker become unavailable to consumers. If a broker dies permanently (e.g., disk failure), all unconsumed data on that node is lost forever. Our goal is to replicate every log segment to multiple broker nodes to improve both the availability and the durability.

We'd like to support the following in Kafka replication:

1. Configurable synchronous and asynchronous replication
2. Small unavailable window (e.g., less than 5 seconds) during broker failures
3. Auto recovery when a failed broker rejoins
4. Balanced load when a broker fails (i.e., the load on the failed broker is evenly spread among multiple surviving brokers)

Here is a complete design proposal for Kafka replication - https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication

Issue Links

• incorporates: KAFKA-352 Throw exception to client if it makes a produce/consume request to a Kafka broker for a topic that hasn't been created
• is blocked by: KAFKA-41 multi-produce and multi-fetch support with replication
• is blocked by: KAFKA-44 Various ZK listeners to support intra-cluster replication
• is blocked by: KAFKA-45 Broker startup, leader election, becoming a leader/follower for intra-cluster replication
• is blocked by: KAFKA-46 Commit thread, ReplicaFetcherThread for intra-cluster replication
• is blocked by: KAFKA-47 Create topic support and new ZK data structures for intra-cluster replication
• is blocked by: KAFKA-48 Implement optional "long poll" support in fetch request
• is blocked by: KAFKA-49 Add acknowledgement to the produce request.
• is blocked by: KAFKA-42 Support rebalancing the partitions with replication
• is blocked by: KAFKA-43 Rebalance to preferred broke with intra-cluster replication support
• is blocked by: KAFKA-174 Add performance suite for Kafka
• is blocked by: KAFKA-330 Add delete topic support
• is blocked by: KAFKA-327 Monitoring and tooling for Kafka replication
• is blocked by: KAFKA-252 Generalize getOffsetsBefore API to a new more general API getLeaderMetadata