Hive
  1. Hive
  2. HIVE-5294

Create collect UDF and make evaluator reusable

    Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: None
    • Labels:
    1. HIVE-5294.1.patch.txt
      18 kB
      Edward Capriolo
    2. HIVE-5294.patch.txt
      18 kB
      Edward Capriolo

      Activity

      Edward Capriolo created issue -
      Edward Capriolo made changes -
      Field Original Value New Value
      Attachment HIVE-5294.patch.txt [ 12603247 ]
      Edward Capriolo made changes -
      Status Open [ 1 ] Patch Available [ 10002 ]
      Hide
      Hive QA added a comment -

      Overall: +1 all checks pass

      Here are the results of testing the latest attachment:
      https://issues.apache.org/jira/secure/attachment/12603247/HIVE-5294.patch.txt

      SUCCESS: +1 3124 tests passed

      Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/752/testReport
      Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/752/console

      Messages:

      Executing org.apache.hive.ptest.execution.PrepPhase
      Executing org.apache.hive.ptest.execution.ExecutionPhase
      Executing org.apache.hive.ptest.execution.ReportingPhase
      

      This message is automatically generated.

      Show
      Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12603247/HIVE-5294.patch.txt SUCCESS: +1 3124 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/752/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/752/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
      Hide
      Brock Noland added a comment -

      This looks good! I wander about the aggregation buffer constructor, specifically:

      Log.error("buffer type was null");

      Won't this lead to a NPE later? If so, should we just throw a RuntimeException?

      Show
      Brock Noland added a comment - This looks good! I wander about the aggregation buffer constructor, specifically: Log.error("buffer type was null"); Won't this lead to a NPE later? If so, should we just throw a RuntimeException?
      Hide
      Edward Capriolo added a comment -

      Yes that should throw at runtime. That was something left over from testing.

      Show
      Edward Capriolo added a comment - Yes that should throw at runtime. That was something left over from testing.
      Edward Capriolo made changes -
      Attachment HIVE-5294.1.patch.txt [ 12603607 ]
      Hide
      Edward Capriolo added a comment -

      .1 throws Runtime exception (which we should never hit anyway)

      Show
      Edward Capriolo added a comment - .1 throws Runtime exception (which we should never hit anyway)
      Hide
      Brock Noland added a comment -

      Agreed. This looks good to me. I plan on committing it if tests pass.

      Show
      Brock Noland added a comment - Agreed. This looks good to me. I plan on committing it if tests pass.
      Hide
      Hive QA added a comment -

      Overall: +1 all checks pass

      Here are the results of testing the latest attachment:
      https://issues.apache.org/jira/secure/attachment/12603607/HIVE-5294.1.patch.txt

      SUCCESS: +1 3126 tests passed

      Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/783/testReport
      Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/783/console

      Messages:

      Executing org.apache.hive.ptest.execution.PrepPhase
      Executing org.apache.hive.ptest.execution.ExecutionPhase
      Executing org.apache.hive.ptest.execution.ReportingPhase
      

      This message is automatically generated.

      Show
      Hive QA added a comment - Overall : +1 all checks pass Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12603607/HIVE-5294.1.patch.txt SUCCESS: +1 3126 tests passed Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/783/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/783/console Messages: Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase This message is automatically generated.
      Hide
      Brock Noland added a comment -

      Committed to trunk! Thank you Edward for your contribution!

      Show
      Brock Noland added a comment - Committed to trunk! Thank you Edward for your contribution!
      Brock Noland made changes -
      Status Patch Available [ 10002 ] Resolved [ 5 ]
      Fix Version/s 0.13.0 [ 12324986 ]
      Resolution Fixed [ 1 ]
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hive-trunk-hadoop2-ptest #103 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/103/)
      HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254)

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java
      • /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q
      • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Show
      Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #103 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/103/ ) HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hive-trunk-hadoop1-ptest #170 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/170/)
      HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254)

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java
      • /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q
      • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Show
      Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #170 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/170/ ) HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Hide
      Sergey Shelukhin added a comment -

      I am getting some build errors that appear to be from this patch. Looks like it might be missing import for ctor, for enum I am not certain and heading out soon... Will look tomorrow if needed.

          [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java:25: package org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMkCollectionEvaluator does not exist
          [javac] import org.apache.hadoop.hive.ql.udf.generic.GenericUDAFMkCollectionEvaluator.BufferType;
          [javac]                                                                              ^
          [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:386: cannot find symbol
          [javac] symbol  : class GenericUDAFCollectList
          [javac] location: class org.apache.hadoop.hive.ql.exec.FunctionRegistry
          [javac]     registerGenericUDAF("collect_list", new GenericUDAFCollectList());
          [javac]                                             ^
          [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java:52: cannot find symbol
          [javac] symbol  : class GenericUDAFMkCollectionEvaluator
          [javac] location: class org.apache.hadoop.hive.ql.udf.generic.GenericUDAFCollectSet
          [javac]     return new GenericUDAFMkCollectionEvaluator(BufferType.SET);
          [javac]                ^
          [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java:52: cannot find symbol
          [javac] symbol  : variable BufferType
          [javac] location: class org.apache.hadoop.hive.ql.udf.generic.GenericUDAFCollectSet
          [javac]     return new GenericUDAFMkCollectionEvaluator(BufferType.SET);
          [javac]                                                 ^
      
      Show
      Sergey Shelukhin added a comment - I am getting some build errors that appear to be from this patch. Looks like it might be missing import for ctor, for enum I am not certain and heading out soon... Will look tomorrow if needed. [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/udf/ generic /GenericUDAFCollectSet.java:25: package org.apache.hadoop.hive.ql.udf. generic .GenericUDAFMkCollectionEvaluator does not exist [javac] import org.apache.hadoop.hive.ql.udf. generic .GenericUDAFMkCollectionEvaluator.BufferType; [javac] ^ [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java:386: cannot find symbol [javac] symbol : class GenericUDAFCollectList [javac] location: class org.apache.hadoop.hive.ql.exec.FunctionRegistry [javac] registerGenericUDAF( "collect_list" , new GenericUDAFCollectList()); [javac] ^ [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/udf/ generic /GenericUDAFCollectSet.java:52: cannot find symbol [javac] symbol : class GenericUDAFMkCollectionEvaluator [javac] location: class org.apache.hadoop.hive.ql.udf. generic .GenericUDAFCollectSet [javac] return new GenericUDAFMkCollectionEvaluator(BufferType.SET); [javac] ^ [javac] /Users/sergey/git/hive3/ql/src/java/org/apache/hadoop/hive/ql/udf/ generic /GenericUDAFCollectSet.java:52: cannot find symbol [javac] symbol : variable BufferType [javac] location: class org.apache.hadoop.hive.ql.udf. generic .GenericUDAFCollectSet [javac] return new GenericUDAFMkCollectionEvaluator(BufferType.SET); [javac] ^
      Hide
      Thejas M Nair added a comment -

      Looks like two files that are part of the patch didn't get added. I will commit those after verifying that it fixes the build.

      Show
      Thejas M Nair added a comment - Looks like two files that are part of the patch didn't get added. I will commit those after verifying that it fixes the build.
      Hide
      Thejas M Nair added a comment -

      Committed the two missing files. The build should work now.

      A         ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
      A         ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
      
      Show
      Thejas M Nair added a comment - Committed the two missing files. The build should work now. A ql/src/java/org/apache/hadoop/hive/ql/udf/ generic /GenericUDAFCollectList.java A ql/src/java/org/apache/hadoop/hive/ql/udf/ generic /GenericUDAFMkCollectionEvaluator.java
      Hide
      Brock Noland added a comment -

      My bad, sorry guys.

      Show
      Brock Noland added a comment - My bad, sorry guys.
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hive-trunk-hadoop2-ptest #104 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/104/)
      HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280)

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
      Show
      Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #104 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/104/ ) HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hive-trunk-hadoop1-ptest #171 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/171/)
      HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280)

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
      Show
      Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #171 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/171/ ) HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hive-trunk-h0.21 #2339 (See https://builds.apache.org/job/Hive-trunk-h0.21/2339/)
      HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280)

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
        HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254)
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java
      • /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q
      • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Show
      Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2339 (See https://builds.apache.org/job/Hive-trunk-h0.21/2339/ ) HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Hide
      Hudson added a comment -

      FAILURE: Integrated in Hive-trunk-hadoop2 #437 (See https://builds.apache.org/job/Hive-trunk-hadoop2/437/)
      HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280)

      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
        HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254)
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
      • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java
      • /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q
      • /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out
      • /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Show
      Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2 #437 (See https://builds.apache.org/job/Hive-trunk-hadoop2/437/ ) HIVE-5294 - Create collect UDF and make evaluator reusable (add missing files) (Edward Capriolo via Brock Noland) (thejas: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524280 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java HIVE-5294 - Create collect UDF and make evaluator reusable (Edward Capriolo via Brock Noland) (brock: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1524254 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectSet.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_collect_set.q /hive/trunk/ql/src/test/results/clientpositive/show_functions.q.out /hive/trunk/ql/src/test/results/clientpositive/udaf_collect_set.q.out
      Swarnim Kulkarni made changes -
      Labels TODOC13
      Transition Time In Source Status Execution Times Last Executer Last Execution Date
      Open Open Patch Available Patch Available
      2m 47s 1 Edward Capriolo 15/Sep/13 20:48
      Patch Available Patch Available Resolved Resolved
      2d 3h 58m 1 Brock Noland 18/Sep/13 00:47

        People

        • Assignee:
          Edward Capriolo
          Reporter:
          Edward Capriolo
        • Votes:
          0 Vote for this issue
          Watchers:
          5 Start watching this issue

          Dates

          • Created:
            Updated:
            Resolved:

            Development