[HIVE-11412] StackOverFlow in SemanticAnalyzer for huge filters (~5000)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Assignee: Hari Sankar Sivarama Subramaniyan
• Reporter: Prasanth Jayachandran
• Created: 30/Jul/15 18:14
• Updated: 08/Aug/15 01:11
• Resolved: 08/Aug/15 01:11

Description

Queries with ~5000 filter conditions fails in SemanticAnalysis

Stack trace:

Exception in thread "main" java.lang.StackOverflowError
	at java.util.HashMap.hash(HashMap.java:366)
	at java.util.HashMap.getEntry(HashMap.java:466)
	at java.util.HashMap.containsKey(HashMap.java:453)
	at org.apache.commons.collections.map.AbstractMapDecorator.containsKey(AbstractMapDecorator.java:83)
	at org.apache.hadoop.conf.Configuration.isDeprecated(Configuration.java:558)
	at org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:605)
	at org.apache.hadoop.conf.Configuration.get(Configuration.java:885)
	at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:907)
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1308)
	at org.apache.hadoop.hive.conf.HiveConf.getBoolVar(HiveConf.java:2641)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.processPositionAlias(SemanticAnalyzer.java:11132)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.processPositionAlias(SemanticAnalyzer.java:11226)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.processPositionAlias(SemanticAnalyzer.java:11226)
	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.processPositionAlias(SemanticAnalyzer.java:11226)

Query:

explain select count(*) from over1k where (
(t=1 and si=2)
or (t=2 and si=3)
or (t=3 and si=4) 
or (t=4 and si=5) 
or (t=5 and si=6) 
or (t=6 and si=7) 
or (t=7 and si=8)
or (t=7 and si=8)
or (t=7 and si=8)
...

Repeat the filter around 5000 times.

Issue Links

• is related to: HIVE-11315 Compilation performance issues for large queries
• is superceded by: HIVE-11398 Parse wide OR and wide AND trees to flat OR/AND trees