Hive
  1. Hive
  2. HIVE-6208

user-defined aggregate functions cannot be used as windowing function

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: UDF
    • Labels:
      None

      Description

      Function registry does a pass to register all GenericUDAFs as window functions. However any aggregate functions added after this (such as a user-added temporary function) don't work as window functions:

      hive> create temporary function mysum as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDAFSum';
      OK
      Time taken: 0.0050 seconds
      hive> explain select mysum(key) over () from src;
      FAILED: NullPointerException null

      java.lang.NullPointerException
      at org.apache.hadoop.hive.ql.parse.PTFTranslator.translate(PTFTranslator.java:354)
      at org.apache.hadoop.hive.ql.parse.PTFTranslator.translate(PTFTranslator.java:194)
      at org.apache.hadoop.hive.ql.parse.WindowingComponentizer.next(WindowingComponentizer.java:86)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genWindowingPlan(SemanticAnalyzer.java:10721)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:7904)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:7862)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:8678)
      at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8904)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:310)
      at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:65)
      at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:310)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:440)
      at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:340)
      at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:996)
      at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1039)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:932)
      at org.apache.hadoop.hive.ql.Driver.run(Driver.java:922)
      at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:268)
      at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:220)
      at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:424)
      at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:792)
      at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:686)
      at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:625)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

        Activity

        Hide
        Jason Dere added a comment -

        Also should not be getting NPE here if the window function cannot be found.

        Show
        Jason Dere added a comment - Also should not be getting NPE here if the window function cannot be found.
        Hide
        Jason Dere added a comment -
        Show
        Jason Dere added a comment - RB at https://reviews.apache.org/r/16921/
        Hide
        Harish Butani added a comment -

        +1

        Show
        Harish Butani added a comment - +1
        Hide
        Hive QA added a comment -

        Overall: +1 all checks pass

        Here are the results of testing the latest attachment:
        https://issues.apache.org/jira/secure/attachment/12623239/HIVE-6208.1.patch

        SUCCESS: +1 4929 tests passed

        Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/925/testReport
        Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/925/console

        Messages:

        Executing org.apache.hive.ptest.execution.PrepPhase
        Executing org.apache.hive.ptest.execution.ExecutionPhase
        Executing org.apache.hive.ptest.execution.ReportingPhase
        

        This message is automatically generated.

        ATTACHMENT ID: 12623239

        Show
        Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12623239/HIVE-6208.1.patch SUCCESS: +1 4929 tests passed Test results: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/925/testReport Console output: http://bigtop01.cloudera.org:8080/job/PreCommit-HIVE-Build/925/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated. ATTACHMENT ID: 12623239
        Hide
        Harish Butani added a comment -

        thanks Jason

        Show
        Harish Butani added a comment - thanks Jason

          People

          • Assignee:
            Jason Dere
            Reporter:
            Jason Dere
          • Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development