[FLINK-22971] Bump Testcontainers to 1.16.0

• Type: Technical Debt
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.12.0
• Fix Version/s: 1.14.0, 1.13.3, 1.12.8
• Component/s: Tests
• Labels: Umbrella auto-deprioritized-critical pull-request-available test-stability
• Assignee: Chesnay Schepler
• Reporter: Xintong Song
• Created: 11/Jun/21 07:19
• Updated: 15/Dec/21 01:40
• Resolved: 07/Sep/21 10:24

Description

Testcontainer versions below 1.16.0 are affected by a race condition in the okhttp transport:
https://github.com/testcontainers/testcontainers-java/issues/3531

The following properties can be observed in many high-frequency test instabilities. They are likely sharing a common root cause.

• Use the Testcontainers framework
• Fail due to no outputs in long time
• A stack similar to the following can be find in the thread dump.
  • Thread is RUNNABLE
  • Testcontainers DockerClient is being created: org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
  • Seems to be downloading something via okhttp

"main" #1 prio=5 os_prio=0 tid=0x00007f4fec00b800 nid=0x4f56 runnable [0x00007f4ff536e000]
   java.lang.Thread.State: RUNNABLE
	at org.testcontainers.shaded.okio.Buffer.getByte(Buffer.java:312)
	at org.testcontainers.shaded.okio.Buffer.readUtf8Line(Buffer.java:766)
	at org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:231)
	at org.testcontainers.shaded.okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:224)
	at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.readChunkSize(Http1ExchangeCodec.java:489)
	at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.read(Http1ExchangeCodec.java:471)
	at org.testcontainers.shaded.okhttp3.internal.Util.skipAll(Util.java:204)
	at org.testcontainers.shaded.okhttp3.internal.Util.discard(Util.java:186)
	at org.testcontainers.shaded.okhttp3.internal.http1.Http1ExchangeCodec$ChunkedSource.close(Http1ExchangeCodec.java:511)
	at org.testcontainers.shaded.okio.ForwardingSource.close(ForwardingSource.java:43)
	at org.testcontainers.shaded.okhttp3.internal.connection.Exchange$ResponseBodySource.close(Exchange.java:313)
	at org.testcontainers.shaded.okio.RealBufferedSource.close(RealBufferedSource.java:476)
	at org.testcontainers.shaded.okhttp3.internal.Util.closeQuietly(Util.java:139)
	at org.testcontainers.shaded.okhttp3.ResponseBody.close(ResponseBody.java:192)
	at org.testcontainers.shaded.okhttp3.Response.close(Response.java:290)
	at org.testcontainers.shaded.com.github.dockerjava.okhttp.OkDockerHttpClient$OkResponse.close(OkDockerHttpClient.java:285)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.lambda$null$0(DefaultInvocationBuilder.java:272)
	at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder$$Lambda$96/1409690119.close(Unknown Source)
	at com.github.dockerjava.api.async.ResultCallbackTemplate.close(ResultCallbackTemplate.java:77)
	at org.testcontainers.utility.ResourceReaper.start(ResourceReaper.java:202)
	at org.testcontainers.DockerClientFactory.client(DockerClientFactory.java:205)
	- locked <0x000000008980f398> (a [Ljava.lang.Object;)
	at org.testcontainers.LazyDockerClient.getDockerClient(LazyDockerClient.java:14)
	at org.testcontainers.LazyDockerClient.listImagesCmd(LazyDockerClient.java:12)
	at org.testcontainers.images.LocalImagesCache.maybeInitCache(LocalImagesCache.java:68)
	- locked <0x000000008980f3e8> (a org.testcontainers.images.LocalImagesCache)
	at org.testcontainers.images.LocalImagesCache.get(LocalImagesCache.java:32)
	at org.testcontainers.images.AbstractImagePullPolicy.shouldPull(AbstractImagePullPolicy.java:18)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:66)
	at org.testcontainers.images.RemoteDockerImage.resolve(RemoteDockerImage.java:27)
	at org.testcontainers.utility.LazyFuture.getResolvedValue(LazyFuture.java:17)
	- locked <0x000000008980f478> (a java.util.concurrent.atomic.AtomicReference)
	at org.testcontainers.utility.LazyFuture.get(LazyFuture.java:39)
	at org.testcontainers.containers.GenericContainer.getDockerImageName(GenericContainer.java:1284)
	at org.testcontainers.containers.GenericContainer.logger(GenericContainer.java:615)
	at org.testcontainers.elasticsearch.ElasticsearchContainer.<init>(ElasticsearchContainer.java:73)
	at org.apache.flink.streaming.connectors.elasticsearch7.ElasticsearchSinkITCase.<clinit>(ElasticsearchSinkITCase.java:48)
	at sun.misc.Unsafe.ensureClassInitialized(Native Method)
	at sun.reflect.UnsafeFieldAccessorFactory.newFieldAccessor(UnsafeFieldAccessorFactory.java:43)
	at sun.reflect.ReflectionFactory.newFieldAccessor(ReflectionFactory.java:156)
	at java.lang.reflect.Field.acquireFieldAccessor(Field.java:1088)
	at java.lang.reflect.Field.getFieldAccessor(Field.java:1069)
	at java.lang.reflect.Field.get(Field.java:393)
	at org.junit.runners.model.FrameworkField.get(FrameworkField.java:73)
	at org.junit.runners.model.TestClass.getAnnotatedFieldValues(TestClass.java:230)
	at org.junit.runners.ParentRunner.classRules(ParentRunner.java:255)
	at org.junit.runners.ParentRunner.withClassRules(ParentRunner.java:244)
	at org.junit.runners.ParentRunner.classBlock(ParentRunner.java:194)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:362)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)

Issue Links

• fixes: FLINK-22387 UpsertKafkaTableITCase hangs when setting up kafka
• fixes: FLINK-22480 KafkaChangelogTableITCase hangs on Azure
• fixes: FLINK-22869 SQLClientSchemaRegistryITCase timeouts on azure
• fixes: FLINK-20329 Elasticsearch7DynamicSinkITCase hangs
• fixes: FLINK-20495 Elasticsearch6DynamicSinkITCase Hang
• fixes: FLINK-20499 ES7 ElasticsearchSinkITCase hangs
• fixes: FLINK-22459 FlinkKafkaProducerITCase testScaleUpAfterScalingDown failed because of Timeout expired after 60000milliseconds while awaiting EndTxn(COMMIT)
• fixes: FLINK-22738 org.apache.flink.connector.jdbc.xa.JdbcExactlyOnceSinkE2eTest no output for 900 seconds due to block on docker container removal
• is related to: FLINK-22386 Introduce cache for docker images
• links to: GitHub Pull Request #17162