[BEAM-12421] Migrate elasticsearchio tests to test containers

• Type: Test
• Status: Resolved
• Priority: P2
• Resolution: Done
• Fix Version/s: 2.31.0
• Component/s: io-java-elasticsearch
• Assignee: Evan Galpin
• Reporter: Etienne Chauchot
• Created: 28/May/21 09:37
• Updated: 01/Sep/21 18:26
• Resolved: 05/Aug/21 14:55

Description

There are more and more flakiness on this IO due to load. I think it is time to migrate from in-jvm ES test framework to test containers.

See discussion: https://lists.apache.org/thread.html/reb68f37c435995a64ded19100e09dfc31c5cf6227feae16494226100%40%3Cdev.beam.apache.org%3E

egalpin I took the liberty to assign to you, feel free to unassign if it is a problem.

lboutros I know you worked on that already, maybe you have a testContainers code for ES to give (I remember you had a starting impl in your github) can you send it to serve as a base for PR ?

To do this migration I think ES2 should be removed. A poll was done among users last year and it concluded to removal of ES 2 (no more supported by elastic since some years)

Issue Links

• relates to: BEAM-5172 org.apache.beam.sdk.io.elasticsearch/ElasticsearchIOTest is flaky
• relates to: BEAM-10485 Failure / flake: ElasticsearchIOTest > testWriteWithIndexFn
• relates to: BEAM-10504 Failure / flake in ElasticSearchIOTest > testWriteFullAddressing and testWriteWithIndexFn
• links to: GitHub Pull Request #14933