[HIVE-28402] Precommit tests fail with OOM when running split-19

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.1.0
• Component/s: Testing Infrastructure
• Labels: pull-request-available
• Assignee: Zhihua Deng
• Reporter: Stamatis Zampetakis
• Created: 25/Jul/24 08:43
• Updated: 29/Jul/24 13:52
• Resolved: 29/Jul/24 13:26

Description

The last 3 runs in master all fail with OOM when running split-19:

• https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2233/pipeline
• https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2234/pipeline
• https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/master/2235/pipeline

[2024-07-25T05:57:46.816Z] [INFO] Running org.apache.hadoop.hive.metastore.client.TestGetPartitions
[2024-07-25T06:00:23.926Z] Exception in thread "Thread-46" java.lang.OutOfMemoryError: GC overhead limit exceeded
[2024-07-25T06:00:23.926Z] 	at java.util.Arrays.copyOfRange(Arrays.java:3664)
[2024-07-25T06:00:23.926Z] 	at java.lang.String.<init>(String.java:207)
[2024-07-25T06:00:23.926Z] 	at java.io.BufferedReader.readLine(BufferedReader.java:356)
[2024-07-25T06:00:24.907Z] 	at java.io.BufferedReader.readLine(BufferedReader.java:389)
[2024-07-25T06:00:24.907Z] 	at org.apache.maven.surefire.shade.common.org.apache.maven.shared.utils.cli.StreamPumper.run(StreamPumper.java:89)
[2024-07-25T06:01:46.664Z] [WARNING] ForkStarter IOException: GC overhead limit exceeded. See the dump file /home/jenkins/agent/workspace/hive-precommit_master/standalone-metastore/metastore-server/target/surefire-reports/2024-07-25T05-50-11_022-jvmRun1.dumpstream
[2024-07-25T06:01:55.003Z] [INFO] Running org.apache.hadoop.hive.metastore.TestFilterHooks
[2024-07-25T06:02:21.747Z] 
[2024-07-25T06:02:21.748Z] Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Thread-49"
[2024-07-25T06:03:08.707Z] [WARNING] ForkStarter IOException: GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded
[2024-07-25T06:03:08.707Z] GC overhead limit exceeded. See the dump file /home/jenkins/agent/workspace/hive-precommit_master/standalone-metastore/metastore-server/target/surefire-reports/2024-07-25T05-50-11_022-jvmRun1.dumpstream
[2024-07-25T06:03:15.362Z] [ERROR] Error closing test event listener:
[2024-07-25T06:03:15.362Z] java.util.concurrent.CompletionException: java.lang.OutOfMemoryError: GC overhead limit exceeded
[2024-07-25T06:03:15.362Z]     at java.util.concurrent.CompletableFuture.encodeThrowable (CompletableFuture.java:273)
[2024-07-25T06:03:15.362Z]     at java.util.concurrent.CompletableFuture.completeThrowable (CompletableFuture.java:280)
[2024-07-25T06:03:15.362Z]     at java.util.concurrent.CompletableFuture$AsyncRun.run (CompletableFuture.java:1643)
[2024-07-25T06:03:15.362Z]     at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
[2024-07-25T06:03:15.362Z]     at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
[2024-07-25T06:03:15.362Z]     at java.lang.Thread.run (Thread.java:748)
[2024-07-25T06:03:15.362Z] Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
[2024-07-25T06:03:15.363Z] [ERROR] GC overhead limit exceeded -> [Help 1]

The OOM is also affecting PR runs.

Issue Links

• is caused by: HIVE-28303 Capture build scans on ge.apache.org to benefit from deep build insights
• is related to: HIVE-28415 Disable Develocity build scans when not used
• links to: GitHub Pull Request #5372