Details
-
New Feature
-
Status: Closed
-
Major
-
Resolution: Fixed
-
None
Description
KIP-36 was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers.
Flink Kafka consumers currently cannot easily use rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in.
This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided.
Attachments
Attachments
Issue Links
Activity
Issue Type | Improvement [ 4 ] | New Feature [ 2 ] |
Resolution | Fixed [ 1 ] | |
Status | In Progress [ 3 ] | Closed [ 6 ] |
Fix Version/s | kafka-3.1.0 [ 12353135 ] |
Remote Link | This issue links to "GitHub Pull Request #53 (Web Link)" [ 332144 ] | This issue links to "GitHub Pull Request #53 (Web Link)" [ 332144 ] |
Remote Link | This issue links to "GitHub Pull Request #53 (Web Link)" [ 332144 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Labels | pull-request-available | pull-request-available stale-assigned |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Status | Open [ 1 ] | In Progress [ 3 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] |
Remote Link | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #20 (Web Link)" [ 315932 ] |
Remote Link | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] |
Remote Link | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] |
Remote Link | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] |
Remote Link | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] |
Labels | pull-request-available |
Remote Link | This issue links to "GitHub Pull Request #22309 (Web Link)" [ 315905 ] |
Description |
[KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awareness+for+Kafka+Streams] was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers.
Flink Kafka consumers currently cannot easily use rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in. This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided. |
[KIP-36|https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment] was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers.
Flink Kafka consumers currently cannot easily use rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in. This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided. |
Assignee | Jeremy DeGroot [ JIRAUSER286379 ] |
Field | Original Value | New Value |
---|---|---|
Description |
[KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awareness+for+Kafka+Streams] was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers.
Flink Kafka consumers currently cannot easily rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in. This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided. |
[KIP-708|https://cwiki.apache.org/confluence/display/KAFKA/KIP-708%3A+Rack+awareness+for+Kafka+Streams] was implemented some time ago in Kafka. This allows brokers and consumers to communicate about the rack (or AWS Availability Zone) they're located in. Reading from a local broker can save money in bandwidth and improve latency for your consumers.
Flink Kafka consumers currently cannot easily use rack awareness if they're deployed across multiple racks or availability zones, because they have no control over which rack the Task Manager they'll be assigned to may be in. This improvement proposes that a Kafka Consumer could be configured with a callback or Future that could be run when it's being configured on the task manager, that will set the appropriate value at runtime if a value is provided. |
Thanks for driving this to the finish line Jeremy DeGroot.
Merged to apache/flink-connector-kafka:main with d89a082180232bb79e3c764228c4e7dbb9eb6b8b