[KAFKA-4462] Improved Kafka Client Compatibility Policy

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.10.1.1
• Fix Version/s: 0.10.2.0
• Component/s: clients
• Labels: kip
• Assignee: Colin McCabe
• Reporter: Colin McCabe
• Reviewer: Jason Gustafson
• Created: 29/Nov/16 18:45
• Updated: 08/Feb/17 18:36
• Resolved: 31/Jan/17 12:17

Description

A proposal to improve the compatibility policy of the Kafka client by supporting the combination of new client, old broker. See https://cwiki.apache.org/confluence/display/KAFKA/KIP-97%3A+Improved+Kafka+Client+RPC+Compatibility+Policy for more details.

Issue Links

• is blocked by: KAFKA-3600 Enhance java clients to use ApiVersion Req/Resp to check if the broker they are talking to supports required api versions
• is related to: SPARK-17344 Kafka 0.8 support for Structured Streaming
• relates to: KAFKA-3308 Use ProtocolVersion during rolling upgrades to decide on a mutually agreed protocol version between brokers, eliminating the need of inter.broker.protocol.version.
• mentioned in: Page Loading...

Sub-Tasks

• Refactor AbstractRequest to contain version information: Resolved
• The client should send older versions of requests to the broker if necessary: Resolved
• Create system tests that run client against older versions of the broker: Resolved
• Add a command to list the broker version information: Resolved
• Add CompatibilityTest to verify that individual features are supported or not by the broker we're connecting to: Resolved
• Implement RecordTooLargeException when communicating with pre-KIP-74 brokers: Resolved
• Client Compatibility follow-up: Resolved
• Update docs/protocol.html with KIP-97 information: Resolved