[IMPALA-1244] Java runs out of heap analyzing large query

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Works for Me
• Affects Version/s: Impala 2.0
• Fix Version/s: Product Backlog

Environment

commit a936c281aa22a542ba4f9b2d2da1160fd35895e1
Author: Skye Wanderman-Milne <[e-mail]>
Date: Fri Sep 12 20:00:14 2014 -0700

    Switch to using FNV hash. Crc doesn't work with different seeds.

• Assignee: casey
• Reporter: casey
• Created: 13/Sep/14 21:58
• Updated: 18/Dec/14 21:12
• Resolved: 18/Dec/14 21:12

Description

The attached very large query causes a java to run out of heap space.

$ impala-shell.sh -d functional -f java_mem_limit.sql
...
ERROR: OutOfMemoryError: Java heap space
...

$ wc -l java_mem_limit.sql 
154 java_mem_limit.sql

stack trace

I0913 14:50:51.327566  7879 jni-util.cc:177] java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2219)
        at java.util.ArrayList.grow(ArrayList.java:242)
        at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:216)
        at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:208)
        at java.util.ArrayList.addAll(ArrayList.java:561)
        at com.cloudera.impala.analysis.WithClause.analyze(WithClause.java:96)
        at com.cloudera.impala.analysis.QueryStmt.analyze(QueryStmt.java:97)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:139)
        at com.cloudera.impala.analysis.WithClause.analyze(WithClause.java:86)
        at com.cloudera.impala.analysis.QueryStmt.analyze(QueryStmt.java:97)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:139)
        at com.cloudera.impala.analysis.WithClause.analyze(WithClause.java:86)
        at com.cloudera.impala.analysis.QueryStmt.analyze(QueryStmt.java:97)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:139)
        at com.cloudera.impala.analysis.InlineViewRef.analyze(InlineViewRef.java:141)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:151)
        at com.cloudera.impala.analysis.UnionStmt$UnionOperand.analyze(UnionStmt.java:74)
        at com.cloudera.impala.analysis.UnionStmt.analyze(UnionStmt.java:152)
        at com.cloudera.impala.analysis.InlineViewRef.analyze(InlineViewRef.java:141)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:151)
        at com.cloudera.impala.analysis.WithClause.analyze(WithClause.java:86)
        at com.cloudera.impala.analysis.QueryStmt.analyze(QueryStmt.java:97)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:139)
        at com.cloudera.impala.analysis.WithClause.analyze(WithClause.java:86)
        at com.cloudera.impala.analysis.QueryStmt.analyze(QueryStmt.java:97)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:139)
        at com.cloudera.impala.analysis.WithClause.analyze(WithClause.java:86)
        at com.cloudera.impala.analysis.QueryStmt.analyze(QueryStmt.java:97)
        at com.cloudera.impala.analysis.SelectStmt.analyze(SelectStmt.java:139)
        at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:284)
        at com.cloudera.impala.analysis.AnalysisContext.analyze(AnalysisContext.java:258)
        at com.cloudera.impala.service.Frontend.analyzeStmt(Frontend.java:698)
I0913 14:50:51.358201  7879 status.cc:44] OutOfMemoryError: Java heap space