[IMPALA-5524] NullPointerException during planning with DISABLE_UNSAFE_SPILLS=TRUE

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: Impala 2.6.0, Impala 2.7.0, Impala 2.8.0
• Fix Version/s: Impala 2.10.0
• Component/s: Frontend
• Labels: newbie
• Epic Color: ghx-label-2
• Assignee: Vincent Tran
• Reporter: Alexander Behm
• Created: 16/Jun/17 06:11
• Updated: 24/Aug/17 16:36
• Resolved: 14/Jul/17 13:09

Description

Setting the DISABLE_UNSAFE_SPILLS query option to TRUE leads to a NullPointerException during query planning if all tables of the query have stats.

Repro:

create table t (i int);
compute stats t;
set disable_unsafe_spills=true;
select * from t;
ERROR: NullPointerException: null

Stack:

I0615 23:06:54.581377 11228 jni-util.cc:176] java.lang.NullPointerException
	at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1027)
	at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1104)
	at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)

The problem is here:

    boolean disableSpilling =
        queryCtx.client_request.query_options.isDisable_unsafe_spills()
          && !queryCtx.tables_missing_stats.isEmpty() <-- tables_missing_stats can be null
          && !analysisResult.getAnalyzer().hasPlanHints();
    queryCtx.setDisable_spilling(disableSpilling);

The reason why "tables_missing_stats" is null can be found in Frontend.createPlanExecInfo():

...
    // Clear pre-existing lists to avoid adding duplicate entries in FE tests.
    queryCtx.unsetTables_missing_stats(); <-- sets to null
    queryCtx.unsetTables_with_corrupt_stats();
    for (TTableName tableName: tablesMissingStats) {
      queryCtx.addToTables_missing_stats(tableName);
    }
    for (TTableName tableName: tablesWithCorruptStats) {
      queryCtx.addToTables_with_corrupt_stats(tableName);
    }
    for (TTableName tableName: tablesWithMissingDiskIds) {
      queryCtx.addToTables_missing_diskids(tableName);
    }
...