[FLINK-31092] Hive ITCases fail with OutOfMemoryError

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 1.17.0, 1.16.1, 1.18.0
• Fix Version/s: 1.17.0, 1.16.2, 1.18.0
• Component/s: Connectors / Hive
• Labels: pull-request-available test-stability
• Assignee: Shengkai Fang
• Reporter: Matthias Pohl
• Created: 15/Feb/23 14:55
• Updated: 06/Mar/23 11:04
• Resolved: 06/Mar/23 11:04

Description

We're experiencing an OutOfMemoryError where the heap space reaches the upper limit:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46161&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=23142

Feb 15 05:05:14 [INFO] Running org.apache.flink.table.catalog.hive.HiveCatalogITCase
Feb 15 05:05:17 [INFO] java.lang.OutOfMemoryError: Java heap space
Feb 15 05:05:17 [INFO] Dumping heap to java_pid9669.hprof ...
Feb 15 05:05:28 [INFO] Heap dump file created [1957090051 bytes in 11.718 secs]
java.lang.OutOfMemoryError: Java heap space
	at org.apache.maven.surefire.booter.ForkedBooter.cancelPingScheduler(ForkedBooter.java:209)
	at org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:419)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:186)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

Issue Links

• is related to: FLINK-30556 Improve the logic for enumerating splits for Hive source to avoid potential OOM
• is related to: FLINK-20945 flink hive insert heap out of memory
• links to: GitHub Pull Request #22072
• links to: GitHub Pull Request #22101
• links to: GitHub Pull Request #22102
• mentioned in: Page Loading...