[HIVE-7271] Speed up unit tests

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0

Release Note

hive.exec.submit.local.task.via.child

Determines whether local tasks (typically mapjoin hashtable generation phase) runs in separate JVM ("true", recommended) or not. Avoids the overhead of spawning new JVM, but can lead to out-of-memory issues.

• Assignee: Gunther Hagleitner
• Reporter: Gunther Hagleitner
• Created: 22/Jun/14 05:22
• Updated: 10/Aug/15 22:37
• Resolved: 25/Jun/14 02:36

Description

Did some experiments to see if there's a way to speed up unit tests. TestCliDriver seemed to take a lot of time just spinning up/tearing down JVMs. I was also curious to see if running everything on a ram disk would help.

Results (I ran tests up to authorization_2):

• Current setup: 40 minutes
• Single JVM (not using child JVM to run all queries): 8 minutes
• Single JVM + ram disk: 7 minutes

So the ram disk didn't help that much. But running tests in single JVM seems worthwhile doing.

Issue Links

• is duplicated by: HIVE-5077 Provide an option to run local task in process
• is related to: HIVE-7924 auto_sortmerge_join_8 sometimes fails with OOM