[KAFKA-50] kafka intra-cluster replication support

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.8.0
• Assignee: Jun Rao
• Reporter: Jun Rao
• Created: 19/Jul/11 21:32
• Updated: 13/Jun/13 15:34
• Resolved: 13/Jun/13 15:34

Description

Currently, Kafka doesn't have replication. Each log segment is stored in a single broker. This limits both the availability and the durability of Kafka. If a broker goes down, all log segments stored on that broker become unavailable to consumers. If a broker dies permanently (e.g., disk failure), all unconsumed data on that node is lost forever. Our goal is to replicate every log segment to multiple broker nodes to improve both the availability and the durability.

We'd like to support the following in Kafka replication:

1. Configurable synchronous and asynchronous replication
2. Small unavailable window (e.g., less than 5 seconds) during broker failures
3. Auto recovery when a failed broker rejoins
4. Balanced load when a broker fails (i.e., the load on the failed broker is evenly spread among multiple surviving brokers)

Here is a complete design proposal for Kafka replication - https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Replication

Issue Links

• incorporates: KAFKA-352 Throw exception to client if it makes a produce/consume request to a Kafka broker for a topic that hasn't been created
• is blocked by: KAFKA-41 multi-produce and multi-fetch support with replication
• is blocked by: KAFKA-44 Various ZK listeners to support intra-cluster replication
• is blocked by: KAFKA-45 Broker startup, leader election, becoming a leader/follower for intra-cluster replication
• is blocked by: KAFKA-46 Commit thread, ReplicaFetcherThread for intra-cluster replication
• is blocked by: KAFKA-47 Create topic support and new ZK data structures for intra-cluster replication
• is blocked by: KAFKA-48 Implement optional "long poll" support in fetch request
• is blocked by: KAFKA-49 Add acknowledgement to the produce request.
• is blocked by: KAFKA-42 Support rebalancing the partitions with replication
• is blocked by: KAFKA-43 Rebalance to preferred broke with intra-cluster replication support
• is blocked by: KAFKA-174 Add performance suite for Kafka
• is blocked by: KAFKA-330 Add delete topic support
• is blocked by: KAFKA-327 Monitoring and tooling for Kafka replication
• is blocked by: KAFKA-252 Generalize getOffsetsBefore API to a new more general API getLeaderMetadata

Sub-Tasks

• implement new producer and consumer request format: Resolved
• Implement an embedded controller: Resolved
• add an admin RPC to communicate state changes between the controller and the broker: Resolved
• upgrade ZKClient to allow conditional updates in ZK: Resolved
• controller failover: Resolved
• using MultiFetch in the follower: Resolved
• Implement clean shutdown in 0.8: Closed
• revisit the broker startup procedure according to V3 design: Resolved
• revisit the become leader and become follower state change operations using V3 design: Resolved
• tie producer-side ack with high watermark and progress of replicas: Resolved
• ZookeeperConsumerConnector needs to connect to new leader after leadership change: Resolved
• remove ZK dependency on producer: Resolved