[HIVE-6922] NullPointerException in collect_set() UDAF

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: UDF
• Assignee: Sun Rui
• Reporter: Sun Rui
• Created: 17/Apr/14 04:03
• Updated: 13/Nov/14 19:44
• Resolved: 24/Apr/14 01:13

Description

Steps to reproduce the bug:

create table temp(key int, value string);
-- leave the table empty
select collect_set(key) from temp where key=0;

Error: java.lang.RuntimeException: Hive Runtime Error while closing operators: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:326)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:471)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:408)
	at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1477)
	at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1141)
	at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:577)
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:318)
	... 7 more
Caused by: java.lang.NullPointerException
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMkCollectionEvaluator.merge(GenericUDAFMkCollectionEvaluator.java:140)
	at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.aggregate(GenericUDAFEvaluator.java:186)
	at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1132)
	... 9 more

The root cause is that in GenericUDAFMkCollectionEvaluator.merge() partialResult could be null but is not validated before it is used.

    List<Object> partialResult = (ArrayList<Object>) internalMergeOI.getList(partial);
    for(Object i : partialResult) {
      putIntoCollection(i, myagg);
    }