[FLINK-17510] StreamingKafkaITCase. testKafka timeouts on downloading Kafka

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.11.3, 1.12.1, 1.13.0
• Fix Version/s: 1.15.0
• Component/s: Build System / Azure Pipelines, Connectors / Kafka, Tests
• Labels: pull-request-available test-stability
• Assignee: Chesnay Schepler
• Reporter: Robert Metzger
• Created: 05/May/20 06:08
• Updated: 17/Dec/21 15:30
• Resolved: 12/Dec/21 12:13

Description

CI: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=585&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5

2020-05-05T00:06:49.7268716Z [INFO] -------------------------------------------------------
2020-05-05T00:06:49.7268938Z [INFO]  T E S T S
2020-05-05T00:06:49.7269282Z [INFO] -------------------------------------------------------
2020-05-05T00:06:50.5336315Z [INFO] Running org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-05-05T00:11:26.8603439Z [ERROR] Tests run: 3, Failures: 0, Errors: 2, Skipped: 0, Time elapsed: 276.323 s <<< FAILURE! - in org.apache.flink.tests.util.kafka.StreamingKafkaITCase
2020-05-05T00:11:26.8604882Z [ERROR] testKafka[1: kafka-version:0.11.0.2](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)  Time elapsed: 120.024 s  <<< ERROR!
2020-05-05T00:11:26.8605942Z java.io.IOException: Process ([wget, -q, -P, /tmp/junit2815750531595874769/downloads/1290570732, https://archive.apache.org/dist/kafka/0.11.0.2/kafka_2.11-0.11.0.2.tgz]) exceeded timeout (120000) or number of retries (3).
2020-05-05T00:11:26.8606732Z 	at org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:132)
2020-05-05T00:11:26.8607321Z 	at org.apache.flink.tests.util.cache.AbstractDownloadCache.getOrDownload(AbstractDownloadCache.java:127)
2020-05-05T00:11:26.8607826Z 	at org.apache.flink.tests.util.cache.LolCache.getOrDownload(LolCache.java:31)
2020-05-05T00:11:26.8608343Z 	at org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.setupKafkaDist(LocalStandaloneKafkaResource.java:98)
2020-05-05T00:11:26.8608892Z 	at org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.before(LocalStandaloneKafkaResource.java:92)
2020-05-05T00:11:26.8609602Z 	at org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
2020-05-05T00:11:26.8610026Z 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-05-05T00:11:26.8610553Z 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-05-05T00:11:26.8610958Z 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-05-05T00:11:26.8611388Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-05-05T00:11:26.8612214Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-05-05T00:11:26.8612706Z 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8613109Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8613551Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8614019Z 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8614442Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8614869Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8615251Z 	at org.junit.runners.Suite.runChild(Suite.java:128)
2020-05-05T00:11:26.8615654Z 	at org.junit.runners.Suite.runChild(Suite.java:27)
2020-05-05T00:11:26.8616060Z 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8616465Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8616893Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8617893Z 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8618490Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8619056Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8619589Z 	at org.junit.runners.Suite.runChild(Suite.java:128)
2020-05-05T00:11:26.8620073Z 	at org.junit.runners.Suite.runChild(Suite.java:27)
2020-05-05T00:11:26.8620745Z 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8621172Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8621585Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8622006Z 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8622403Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8622811Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8623212Z 	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
2020-05-05T00:11:26.8623698Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
2020-05-05T00:11:26.8635914Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
2020-05-05T00:11:26.8636538Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
2020-05-05T00:11:26.8637056Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
2020-05-05T00:11:26.8637543Z 	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
2020-05-05T00:11:26.8638072Z 	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2020-05-05T00:11:26.8638590Z 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2020-05-05T00:11:26.8639065Z 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2020-05-05T00:11:26.8639502Z 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2020-05-05T00:11:26.8639902Z Caused by: java.io.IOException: Process failed due to timeout.
2020-05-05T00:11:26.8640420Z 	at org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:114)
2020-05-05T00:11:26.8641189Z 	at org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:125)
2020-05-05T00:11:26.8641598Z 	... 42 more
2020-05-05T00:11:26.8641915Z 
2020-05-05T00:11:26.8642783Z [ERROR] testKafka[2: kafka-version:2.2.2](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)  Time elapsed: 120.004 s  <<< ERROR!
2020-05-05T00:11:26.8643829Z java.io.IOException: Process ([wget, -q, -P, /tmp/junit4015087902971183123/downloads/-1713883086, https://archive.apache.org/dist/kafka/2.2.2/kafka_2.11-2.2.2.tgz]) exceeded timeout (120000) or number of retries (3).
2020-05-05T00:11:26.8644556Z 	at org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:132)
2020-05-05T00:11:26.8645138Z 	at org.apache.flink.tests.util.cache.AbstractDownloadCache.getOrDownload(AbstractDownloadCache.java:127)
2020-05-05T00:11:26.8645623Z 	at org.apache.flink.tests.util.cache.LolCache.getOrDownload(LolCache.java:31)
2020-05-05T00:11:26.8646140Z 	at org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.setupKafkaDist(LocalStandaloneKafkaResource.java:98)
2020-05-05T00:11:26.8646720Z 	at org.apache.flink.tests.util.kafka.LocalStandaloneKafkaResource.before(LocalStandaloneKafkaResource.java:92)
2020-05-05T00:11:26.8647201Z 	at org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
2020-05-05T00:11:26.8647610Z 	at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
2020-05-05T00:11:26.8648025Z 	at org.junit.rules.RunRules.evaluate(RunRules.java:20)
2020-05-05T00:11:26.8648401Z 	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
2020-05-05T00:11:26.8648823Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
2020-05-05T00:11:26.8649310Z 	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
2020-05-05T00:11:26.8649736Z 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8650123Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8650560Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8650968Z 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8651367Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8652115Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8652517Z 	at org.junit.runners.Suite.runChild(Suite.java:128)
2020-05-05T00:11:26.8654347Z 	at org.junit.runners.Suite.runChild(Suite.java:27)
2020-05-05T00:11:26.8654809Z 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8655235Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8655692Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8656141Z 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8656544Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8656956Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8657353Z 	at org.junit.runners.Suite.runChild(Suite.java:128)
2020-05-05T00:11:26.8657727Z 	at org.junit.runners.Suite.runChild(Suite.java:27)
2020-05-05T00:11:26.8658110Z 	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
2020-05-05T00:11:26.8658525Z 	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
2020-05-05T00:11:26.8658923Z 	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
2020-05-05T00:11:26.8659343Z 	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
2020-05-05T00:11:26.8659740Z 	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
2020-05-05T00:11:26.8660148Z 	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
2020-05-05T00:11:26.8660740Z 	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
2020-05-05T00:11:26.8661213Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
2020-05-05T00:11:26.8661734Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107)
2020-05-05T00:11:26.8662334Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83)
2020-05-05T00:11:26.8662808Z 	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
2020-05-05T00:11:26.8663278Z 	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
2020-05-05T00:11:26.8663817Z 	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
2020-05-05T00:11:26.8664315Z 	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
2020-05-05T00:11:26.8664791Z 	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
2020-05-05T00:11:26.8665264Z 	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
2020-05-05T00:11:26.8665735Z Caused by: java.io.IOException: Process failed due to timeout.
2020-05-05T00:11:26.8667503Z 	at org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:114)
2020-05-05T00:11:26.8668023Z 	at org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlockingWithRetry(AutoClosableProcess.java:125)
2020-05-05T00:11:26.8668347Z 	... 42 more
2020-05-05T00:11:26.8668440Z 
2020-05-05T00:11:26.8671270Z [INFO] Running org.apache.flink.tests.util.kafka.SQLClientKafkaITCase

Issue Links

• is duplicated by: FLINK-21796 SQLClientKafkaITCase.testKafka fails to download Kafka
• is related to: FLINK-19158 Revisit java e2e download timeouts
• is related to: FLINK-17400 LocalStandaloneKafkaResource.setupKafkaDist fails due to download timeout
• relates to: FLINK-20057 Use docker images instead of direct downloads in all end to end tests
• links to: GitHub Pull Request #18105