[HIVE-6984] Analyzing partitioned table with NULL values for the partition column failed with NPE

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.0
• Component/s: Statistics
• Assignee: Xuefu Zhang
• Reporter: Xuefu Zhang
• Created: 28/Apr/14 23:08
• Updated: 13/Nov/14 19:40
• Resolved: 01/May/14 23:07

Description

The following describes how to produce the bug:

hive> desc test2;
name                	string              	                    
age                 	int                 	                    

hive> select * from test2;
6666666666666666666	NULL
5555555555555555555	NULL
tom	15
john	NULL
mayr	40
	30
	NULL

hive> create table test3(name string) partitioned by (age int);

hive> from test2 insert overwrite table test3 partition(age) select test2.name, test2.age;
Loading data to table default.test3 partition (age=null)
	Loading partition {age=40}
	Loading partition {age=__HIVE_DEFAULT_PARTITION__}
	Loading partition {age=30}
	Loading partition {age=15}
Partition default.test3{age=15} stats: [numFiles=1, numRows=1, totalSize=4, rawDataSize=3]
Partition default.test3{age=30} stats: [numFiles=1, numRows=1, totalSize=1, rawDataSize=0]
Partition default.test3{age=40} stats: [numFiles=1, numRows=1, totalSize=5, rawDataSize=4]
Partition default.test3{age=__HIVE_DEFAULT_PARTITION__} stats: [numFiles=1, numRows=4, totalSize=46, rawDataSize=42]

hive> analyze table test3 partition(age) compute statistics;
...
Task with the most failures(4): 
-----
Diagnostic Messages for this Task:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
	at org.apache.hado

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask

The following is the stack trace in mapper log:

2014-04-28 15:39:25,073 FATAL org.apache.hadoop.hive.ql.exec.mr.ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"name":"6666666666666666666","age":null,"raw__data__size":19}
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
	at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:177)
	at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
	at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
	at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
	at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.gatherStats(TableScanOperator.java:149)
	at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:90)
	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:539)
	... 9 more