[HIVE-21072] NPE when running partitioned CTAS statements

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: L. C. Hsieh
• Created: 27/Dec/18 05:10
• Updated: 27/Feb/24 22:24

Description

HIVE-20241 adds support of partitioned CTAS statements:

CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
SELECT value, key FROM src where key > 200 and key < 300;

 
However, I've tried this feature by checking out latest branch-3, and encountered NPE:

hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
FAILED: NullPointerException null

I also ran the query test partition_ctas.q. The test passes when using TestMiniLlapLocalCliDriver, but when I go to test it with TestCliDriver manually, it also throws NullPointerException:

2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main] ql.Driver: FAILED: NullPointerException null
java.lang.NullPointerException
    at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(GenMapRedUtils.java:2103)
    at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(GenMapRedUtils.java:1323)
    at org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:113)
    at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
    at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:54)
    at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
    at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
    at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:65)
    at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
    at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(MapReduceCompiler.java:323)
    at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(TaskCompiler.java:244)
    at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12503)
    at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:357)
    at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
    at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:166)
    at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:285)
    at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:664)
    at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1854)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1801)
    at org.apache.hadoop.hive.ql.Driver.compileAndRespond(Driver.java:1796)
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(ReExecDriver.java:126)
    at org.apache.hadoop.hive.ql.reexec.ReExecDriver.run(ReExecDriver.java:214)
    at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:239)
    at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:188)
    at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:402)

Issue Links

• relates to: HIVE-20707 Automatic partition management
• links to: GitHub Pull Request #510
• links to: GitHub Pull Request #510