[FLINK-34403] VeryBigPbProtoToRowTest#testSimple cannot pass due to OOM

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.20.0
• Fix Version/s: 1.20.0
• Component/s: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
• Labels: pull-request-available test-stability
• Assignee: Matthias Pohl
• Reporter: Benchao Li
• Created: 07/Feb/24 06:20
• Updated: 14/Feb/24 16:56
• Resolved: 14/Feb/24 16:56

Description

After FLINK-33611 merged, the misc test on GHA cannot pass due to out of memory error, throwing following exceptions:

Error: 05:43:21 05:43:21.768 [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 40.98 s <<< FAILURE! -- in org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest
Error: 05:43:21 05:43:21.773 [ERROR] org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple -- Time elapsed: 40.97 s <<< ERROR!
Feb 07 05:43:21 org.apache.flink.util.FlinkRuntimeException: Error in serialization.
Feb 07 05:43:21 	at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:327)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:162)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.graph.StreamGraph.getJobGraph(StreamGraph.java:1007)
Feb 07 05:43:21 	at org.apache.flink.client.StreamGraphTranslator.translateToJobGraph(StreamGraphTranslator.java:56)
Feb 07 05:43:21 	at org.apache.flink.client.FlinkPipelineTranslationUtil.getJobGraph(FlinkPipelineTranslationUtil.java:45)
Feb 07 05:43:21 	at org.apache.flink.client.deployment.executors.PipelineExecutorUtils.getJobGraph(PipelineExecutorUtils.java:61)
Feb 07 05:43:21 	at org.apache.flink.client.deployment.executors.LocalExecutor.getJobGraph(LocalExecutor.java:104)
Feb 07 05:43:21 	at org.apache.flink.client.deployment.executors.LocalExecutor.execute(LocalExecutor.java:81)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2440)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.executeAsync(StreamExecutionEnvironment.java:2421)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.datastream.DataStream.executeAndCollectWithClient(DataStream.java:1495)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1382)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.datastream.DataStream.executeAndCollect(DataStream.java:1367)
Feb 07 05:43:21 	at org.apache.flink.formats.protobuf.ProtobufTestHelper.validateRow(ProtobufTestHelper.java:66)
Feb 07 05:43:21 	at org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:89)
Feb 07 05:43:21 	at org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:76)
Feb 07 05:43:21 	at org.apache.flink.formats.protobuf.ProtobufTestHelper.rowToPbBytes(ProtobufTestHelper.java:71)
Feb 07 05:43:21 	at org.apache.flink.formats.protobuf.VeryBigPbRowToProtoTest.testSimple(VeryBigPbRowToProtoTest.java:37)
Feb 07 05:43:21 	at java.lang.reflect.Method.invoke(Method.java:498)
Feb 07 05:43:21 Caused by: java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: Self-suppression not permitted
Feb 07 05:43:21 	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
Feb 07 05:43:21 	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.createJobGraph(StreamingJobGraphGenerator.java:323)
Feb 07 05:43:21 	... 18 more
Feb 07 05:43:21 Caused by: java.lang.IllegalArgumentException: Self-suppression not permitted
Feb 07 05:43:21 	at java.lang.Throwable.addSuppressed(Throwable.java:1072)
Feb 07 05:43:21 	at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:556)
Feb 07 05:43:21 	at org.apache.flink.util.InstantiationUtil.writeObjectToConfig(InstantiationUtil.java:486)
Feb 07 05:43:21 	at org.apache.flink.streaming.api.graph.StreamConfig.lambda$triggerSerializationAndReturnFuture$0(StreamConfig.java:182)
Feb 07 05:43:21 	at java.util.concurrent.CompletableFuture.uniAccept(CompletableFuture.java:670)
Feb 07 05:43:21 	at java.util.concurrent.CompletableFuture$UniAccept.tryFire(CompletableFuture.java:646)
Feb 07 05:43:21 	at java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
Feb 07 05:43:21 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
Feb 07 05:43:21 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
Feb 07 05:43:21 	at java.lang.Thread.run(Thread.java:750)
Feb 07 05:43:21 Caused by: java.lang.OutOfMemoryError: Java heap space

See more details : https://github.com/apache/flink/actions/runs/7810182427/job/21303415607

Issue Links

• is duplicated by: FLINK-34408 VeryBigPbProtoToRowTest#testSimple fails with OOM
• relates to: FLINK-33611 Support Large Protobuf Schemas
• links to: GitHub Pull Request #24302
• links to: GitHub Pull Request #24311