[BEAM-6279] Failures in ElasticSearchIOTest: testWriteRetryValidRequest (versions 2 & 5) testDefaultRetryPredicate (version 5)

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: Not applicable
• Component/s: io-java-elasticsearch, test-failures
• Labels: flake
• Assignee: Etienne Chauchot
• Reporter: Kenneth Knowles
• Created: 20/Dec/18 15:05
• Updated: 16/May/20 13:29
• Resolved: 16/Jan/19 15:03

Description

https://builds.apache.org/job/beam_PreCommit_Java_Cron/730/

https://scans.gradle.com/s/6rpbvgwx2nk7c

I don't see recent changes, so thinking this is a flake.

testDefaultRetryPredicate:

java.lang.AssertionError
: 
All incoming requests on node [node_s0] should have finished. Expected 0 but got 2540 Expected: <0L> but: was <2540L>
at __randomizedtesting.SeedInfo.seed([4CF46FDC92DC07B8:B39A5DDFBC85274E]:0)
at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
at org.junit.Assert.assertThat(Assert.java:956)
at org.elasticsearch.test.InternalTestCluster.lambda$assertRequestsFinished$17(InternalTestCluster.java:2086)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:705)
at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:679)
at org.elasticsearch.test.InternalTestCluster.assertRequestsFinished(InternalTestCluster.java:2083)
at org.elasticsearch.test.InternalTestCluster.assertAfterTest(InternalTestCluster.java:2062)
at org.elasticsearch.test.ESIntegTestCase.afterInternal(ESIntegTestCase.java:581)
at org.elasticsearch.test.ESIntegTestCase.cleanUpCluster(ESIntegTestCase.java:2054)
...

This looks like an internal assertion (aka checkState) failed and the cluster became unhealthy, maybe?

 testWriteRetryValidRequest:

org.apache.beam.sdk.Pipeline$PipelineExecutionException
: 
java.io.IOException: listener timeout after waiting for [90000] ms
Open stacktrace
Caused by: 
java.io.IOException
: 
listener timeout after waiting for [90000] ms
Close stacktrace
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:899)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:227)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:321)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.flushBatch(ElasticsearchIO.java:1285)
at org.apache.beam.sdk.io.elasticsearch.ElasticsearchIO$Write$WriteFn.finishBundle(ElasticsearchIO.java:1260)

Issue Links

• links to: GitHub Pull Request #7535