Uploaded image for project: 'Hivemall'
  1. Hivemall
  2. HIVEMALL-157

Null Pointer Exception in to_ordered_list UDAF for empty (partial) result

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.5.0
    • Labels:
      None

      Description

      Consider the following query:

      SELECT to_ordered_list(null, null)
      

      Here, even though to_ordered_list() automatically ignores NULL key and/or value, the query throws NPE because of NULL queue handler access in drainQueue():

      org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
              at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1102)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:683)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
              at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:697)
              at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:189)
              at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
              at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
              at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:270)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
              at java.util.concurrent.FutureTask.run(FutureTask.java:266)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
              at java.lang.Thread.run(Thread.java:745)
      Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.NullPointerException
              at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1060)
              at org.apache.hadoop.hive.ql.exec.GroupByOperator.closeOp(GroupByOperator.java:1099)
              ... 14 more
      Caused by: java.lang.NullPointerException
              at hivemall.tools.list.UDAFToOrderedList$UDAFToOrderedListEvaluator$QueueAggregationBuffer.drainQueue(UDAFToOrderedList.java:411)
              at hivemall.tools.list.UDAFToOrderedList$UDAFToOrderedListEvaluator.terminatePartial(UDAFToOrderedList.java:313)
              at org.apache.hadoop.hive.ql.udf.generic.GenericUDAFEvaluator.evaluate(GenericUDAFEvaluator.java:204)
              at org.apache.hadoop.hive.ql.exec.GroupByOperator.forward(GroupByOperator.java:1020)
              at org.apache.hadoop.hive.ql.exec.GroupByOperator.flush(GroupByOperator.java:1043)
              ... 15 more
      

      Since iterate() accepts NULL inputs, drainQueue() should not fail; it has to return something like empty list.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                takuti Takuya Kitazawa
                Reporter:
                takuti Takuya Kitazawa
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: