[FLINK-23977] Add Support for Dynamic Elastic Cluster/Index Routing

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Won't Do
• Component/s: Connectors / ElasticSearch
• Labels: elasticsearch pull-request-available
• Assignee: Rion Williams
• Reporter: Rion Williams
• Created: 25/Aug/21 19:02
• Updated: 17/Nov/21 15:21
• Resolved: 09/Oct/21 19:29

Description

Currently only index-level dynamic routing is supported within the Elasticsearch connector, which can be problematic if information such as the host of the cluster (e.g. HttpHosts) is not known until run-time. 

The idea behind this ticket would be to implement a wrapper that would allow Elasticsearch sink instances to be created at run time and stored within an in-memory cache (i.e. HashMap) using a key-value association where the key represents a deterministically unique "route" and the value is the sink to which all items with a given key are written.

Issue Links

• links to: GitHub Pull Request #17061