[KAFKA-9876] Implement Raft Protocol for Metadata Quorum

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: core
• Assignee: Jason Gustafson
• Reporter: Jason Gustafson
• Created: 15/Apr/20 21:26
• Updated: 18/Nov/20 06:40

Description

This tracks the completion of the Raft Protocol specified in KIP-595: https://cwiki.apache.org/confluence/display/KAFKA/KIP-595%3A+A+Raft+Protocol+for+the+Metadata+Quorum. If/when the KIP is approved by the community, we will create smaller sub-tasks to track overall prgress.

Issue Links

• mentioned in: Page Loading...

Sub-Tasks

• Fetch protocol changes for KIP-595: Resolved
• Fix edge case in Raft truncation protocol: Resolved
• Core Raft implementation: Resolved
• Fix minor behavior difference in `MockLog`: Resolved
• Unit tests for VotedState: Resolved
• Explore performance impact of leader fsync deferral: Open
• Voters should always initialize as followers: Resolved
• Add log flush semantics to simulation test: Resolved
• Enhance Raft exception handling: Open
• Add linger semantics to raft: Resolved
• Raft client should push all committed data to listeners: Resolved
• LeaderChangeMessage should include the leader as one of the voters: Resolved
• Bypass log validation for writes to raft log: Resolved
• Add resigned state to raft state machine to preserve leader/epoch information: Resolved
• Fix VotedToUnattached test error: Resolved
• Raft leader should flush accumulated writes after a min size is reached: Resolved
• Raft leader should resign after write failures: Open
• Decide whether Raft listener callback errors are fatal: Open
• Complete fetches in purgatory immediately after raft leader resigns: Resolved
• Add missing unit test for `UnattachedState`: Resolved
• Add clusterId validation to Fetch handling: Resolved
• Ensure raft io thread wakes up after linger expiration: Resolved
• Consider using "acknowledged" over "endorsing" for voters which have recognized the current leader: Resolved
• Refactor raft outbound request channel: Resolved
• Consider better return type of RaftClient.scheduleAppend: Resolved
• Raft observers should not require an id to fetch: Resolved
• Loosen monotonic fetch offset validation by raft leader: Resolved
• Support non-routable quorum voter addresses: Resolved
• Change the BatchAccumulator to split records into batches: Resolved
• KafkaRaftClient should use the LEO when appending LeaderChangeMessage: Resolved
• KafkaMetadataLog and MockLock should validate that appended epochs are monotonically: Open
• Improve KafkaRaftClient handling of graceful shutdown: Open
• ClusterId validation for Vote, BeginQourum and EndQuorum: Resolved
• Improve raft simulation tests: Resolved
• Raft should prevent truncation below high watermark: Resolved
• Extend LogManager to cover the metadata topic: Open
• Decide whether inconsistent cluster id error are fatal: Open
• Allow votes to be granted in resigned state: Resolved
• Support api to resign raft leadership: Resolved
• Use UncheckedIOException when wrapping IOException: Resolved
• Add integration tests for "test-kraft-server-start": Open
• Notify listeners of raft client shutdowns: Open
• Make kraft configuration properties public: Resolved
• Remove the method RaftClient.leaderAndEpoch: Open
• Revisit the usage of BufferSuppliers in Kraft: Open
• Validate node id, process role and quorum voters: Resolved
• Notify listener of leader change on registration: Open
• Resign leader on network partition: Resolved
• KRaft Obsever are not required to flush on every append: Resolved