[BEAM-2488] Elasticsearch IO should read also in replica shards

• Type: Bug
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.1.0
• Component/s: io-java-elasticsearch
• Assignee: Etienne Chauchot
• Reporter: Etienne Chauchot
• Created: 21/Jun/17 07:39
• Updated: 16/May/20 13:46
• Resolved: 28/Jun/17 08:11

Description

To avoid duplication of data ElasticsearchIO reads from primary shards only and filters out replica shards. But in reality, even if _shard-preference:shardId is set in scroll request, ES internally load balances requests between primary and replica shards and ensures that there will be no duplicates. Targeting all the shards and letting ES deal with replicas is better in some corner cases like failover.

Issue Links

• links to: GitHub Pull Request #3410