[FLINK-25924] KDF Integration tests intermittently fails

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Fix Version/s: 1.15.0
• Component/s: Connectors / Common
• Labels: pull-request-available test-stability
• Assignee: Ahmed Hamdy
• Reporter: Zichen Liu
• Created: 02/Feb/22 11:35
• Updated: 07/Apr/22 05:35
• Resolved: 07/Feb/22 10:00

Description

Intermittent failures introduced as part of merge (PR#18314: FLINK-24228[connectors/firehose] - Unified Async Sink for Kinesis Firehose):

1. Failures are intermittent and affecting c. 1 in 7 of builds- on flink-ci.flink and flink-ci.flink-master-mirror .
2. The issue looks identical to the KinesaliteContainer startup issue (Appendix 1).
3. I have managed to reproduce the issue locally - if I start some parallel containers and keep them running - and then run KinesisFirehoseSinkITCase  then c. 1 in 6 gives the error.
4. The errors have a slightly different appearance on flink-ci.flink-master-mirror vs flink-ci.flink which has the same appearance as local. I only hope it is a difference in logging/killing environment variables. (and that there aren’t 2 distinct issues)

Appendix 1:

org.testcontainers.containers.ContainerLaunchException: Container startup failed

at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:336)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)
at org.testcontainers.containers.GenericContainer.starting(GenericContainer.java:1066)
at 
... 11 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:525)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
... 12 more
Caused by: org.rnorth.ducttape.TimeoutException: Timeout waiting for result with exception
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:54)
at

Issue Links

• is cloned by: FLINK-25948 KDS / KDF Sink should call .close() to clean up resources
• is duplicated by: FLINK-25922 KinesisFirehoseSinkITCase hangs on AZP
• is related to: FLINK-24228 [FLIP-171] Firehose implementation of Async Sink
• is superceded by: FLINK-25944 Intermittent Failures on KDF AZP
• relates to: FLINK-25945 Intermittent Failures on KDF AZP 2
• relates to: FLINK-24433 "No space left on device" in Azure e2e tests
• supercedes: FLINK-25922 KinesisFirehoseSinkITCase hangs on AZP
• links to: GitHub Pull Request #18606
• links to: GitHub Pull Request #18633