Hive
  1. Hive
  2. HIVE-5279

Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.13.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how previous serialization mechanism solved this but, kryo complaints that it's not Serializable and fails the query.

      The log below is the example,

      java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
      Serialization trace:
      inputOI (org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
      genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
      aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
      conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
      childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
      childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
      aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
      	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
      	at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
      	at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
      	at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
      	at org.apache.h
      

      If this cannot be fixed in somehow, some UDAFs should be modified to be run on hive-0.13.0

      1. D12963.5.patch
        28 kB
        Phabricator
      2. D12963.4.patch
        24 kB
        Phabricator
      3. D12963.3.patch
        12 kB
        Phabricator
      4. D12963.2.patch
        12 kB
        Phabricator
      5. D12963.1.patch
        10 kB
        Phabricator
      6. 5279.patch
        6 kB
        Ashutosh Chauhan

        Activity

        Hide
        Hudson added a comment -

        ABORTED: Integrated in Hive-trunk-hadoop2 #457 (See https://builds.apache.org/job/Hive-trunk-hadoop2/457/)
        HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
        • /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Show
        Hudson added a comment - ABORTED: Integrated in Hive-trunk-hadoop2 #457 (See https://builds.apache.org/job/Hive-trunk-hadoop2/457/ ) HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-h0.21 #2358 (See https://builds.apache.org/job/Hive-trunk-h0.21/2358/)
        HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
        • /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-h0.21 #2358 (See https://builds.apache.org/job/Hive-trunk-h0.21/2358/ ) HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop2-ptest #115 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/115/)
        HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
        • /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop2-ptest #115 (See https://builds.apache.org/job/Hive-trunk-hadoop2-ptest/115/ ) HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hive-trunk-hadoop1-ptest #181 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/181/)
        HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117)

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
        • /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        • /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q
        • /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml
        • /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Show
        Hudson added a comment - FAILURE: Integrated in Hive-trunk-hadoop1-ptest #181 (See https://builds.apache.org/job/Hive-trunk-hadoop1-ptest/181/ ) HIVE-5279 : Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc (Navis via Ashutosh Chauhan) (hashutosh: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1526117 ) /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java /hive/trunk/ql/src/test/queries/clientpositive/udaf_sum_list.q /hive/trunk/ql/src/test/results/clientpositive/udaf_sum_list.q.out /hive/trunk/ql/src/test/results/compiler/plan/groupby1.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby2.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby3.q.xml /hive/trunk/ql/src/test/results/compiler/plan/groupby5.q.xml
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Navis!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis!
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc".

        Fixed tests

        Reviewers: ashutoshc, JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D12963

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D12963?vs=40299&id=40365#toc

        BRANCH
        HIVE-5279

        ARCANIST PROJECT
        hive

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java
        ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java
        ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        ql/src/test/queries/clientpositive/udaf_sum_list.q
        ql/src/test/results/clientpositive/udaf_sum_list.q.out
        ql/src/test/results/compiler/plan/groupby1.q.xml
        ql/src/test/results/compiler/plan/groupby2.q.xml
        ql/src/test/results/compiler/plan/groupby3.q.xml
        ql/src/test/results/compiler/plan/groupby5.q.xml

        To: JIRA, ashutoshc, navis

        Show
        Phabricator added a comment - navis updated the revision " HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc". Fixed tests Reviewers: ashutoshc, JIRA REVISION DETAIL https://reviews.facebook.net/D12963 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D12963?vs=40299&id=40365#toc BRANCH HIVE-5279 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCollectList.java ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFMkCollectionEvaluator.java ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java ql/src/test/queries/clientpositive/udaf_sum_list.q ql/src/test/results/clientpositive/udaf_sum_list.q.out ql/src/test/results/compiler/plan/groupby1.q.xml ql/src/test/results/compiler/plan/groupby2.q.xml ql/src/test/results/compiler/plan/groupby3.q.xml ql/src/test/results/compiler/plan/groupby5.q.xml To: JIRA, ashutoshc, navis
        Hide
        Ashutosh Chauhan added a comment -

        Quite a few tests failed. 35 in TestCliDriver & 2 in TestNegativeCliDriver, including database_drop.q,index*.q,ql_rewrite_gbtoidx.q,show_indexes_edge_cases.q,show_indexes_syntax.q,udaf_collect_set.q,union_view.q,virtual_column.q. In -ve testcases, index_compact_entry_limit.q,index_compact_size_limit.q

        Show
        Ashutosh Chauhan added a comment - Quite a few tests failed. 35 in TestCliDriver & 2 in TestNegativeCliDriver, including database_drop.q,index*.q,ql_rewrite_gbtoidx.q,show_indexes_edge_cases.q,show_indexes_syntax.q,udaf_collect_set.q,union_view.q,virtual_column.q. In -ve testcases, index_compact_entry_limit.q,index_compact_size_limit.q
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc".

        Fixed test fails

        Reviewers: ashutoshc, JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D12963

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D12963?vs=40065&id=40299#toc

        BRANCH
        HIVE-5279

        ARCANIST PROJECT
        hive

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        ql/src/test/queries/clientpositive/udaf_sum_list.q
        ql/src/test/results/clientpositive/udaf_sum_list.q.out
        ql/src/test/results/compiler/plan/groupby1.q.xml
        ql/src/test/results/compiler/plan/groupby2.q.xml
        ql/src/test/results/compiler/plan/groupby3.q.xml
        ql/src/test/results/compiler/plan/groupby5.q.xml

        To: JIRA, ashutoshc, navis

        Show
        Phabricator added a comment - navis updated the revision " HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc". Fixed test fails Reviewers: ashutoshc, JIRA REVISION DETAIL https://reviews.facebook.net/D12963 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D12963?vs=40065&id=40299#toc BRANCH HIVE-5279 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java ql/src/test/queries/clientpositive/udaf_sum_list.q ql/src/test/results/clientpositive/udaf_sum_list.q.out ql/src/test/results/compiler/plan/groupby1.q.xml ql/src/test/results/compiler/plan/groupby2.q.xml ql/src/test/results/compiler/plan/groupby3.q.xml ql/src/test/results/compiler/plan/groupby5.q.xml To: JIRA, ashutoshc, navis
        Hide
        Navis added a comment -

        Missed those tests and sorry for delay.

        Show
        Navis added a comment - Missed those tests and sorry for delay.
        Hide
        Brock Noland added a comment -

        Yeah I agree. I think it's probably best just to start providing a no-arg constructor for things that need to be serialized. That method can be protected or private.

        Show
        Brock Noland added a comment - Yeah I agree. I think it's probably best just to start providing a no-arg constructor for things that need to be serialized. That method can be protected or private.
        Hide
        Edward Capriolo added a comment -

        I just solved a similar issue here:
        https://issues.apache.org/jira/browse/HIVE-5294

        We are probably best off turning things that need to be serialized into JavaBean like classe no-arg constructor setters + getters. I know that is not always the most elegant way, but at least it would be consistent, when you dive around the code best you usually get tripped up by some weird construct with a static initializer, or something.

        Show
        Edward Capriolo added a comment - I just solved a similar issue here: https://issues.apache.org/jira/browse/HIVE-5294 We are probably best off turning things that need to be serialized into JavaBean like classe no-arg constructor setters + getters. I know that is not always the most elegant way, but at least it would be consistent, when you dive around the code best you usually get tripped up by some weird construct with a static initializer, or something.
        Hide
        Leo Romanoff added a comment -

        Regarding the issue with Kryo's exception about no-arg constructor:
        Please read this section: https://code.google.com/p/kryo/#Object_creation

        Basically, you can force Kryo to create objects without invoking any constructor at all, but you should be careful with it and your class should be written in such a way that it can handle it.

        -Leo

        Show
        Leo Romanoff added a comment - Regarding the issue with Kryo's exception about no-arg constructor: Please read this section: https://code.google.com/p/kryo/#Object_creation Basically, you can force Kryo to create objects without invoking any constructor at all, but you should be careful with it and your class should be written in such a way that it can handle it. -Leo
        Hide
        Ashutosh Chauhan added a comment -

        With latest patch tests in TestCliDriver & TestContribCliDriver tests are no more failing, so thats a progress. However, 4 tests in TestParse are still failing, as previously, namely groupby1.q groupby2.q groupby3.q and groupby5.q

        Show
        Ashutosh Chauhan added a comment - With latest patch tests in TestCliDriver & TestContribCliDriver tests are no more failing, so thats a progress. However, 4 tests in TestParse are still failing, as previously, namely groupby1.q groupby2.q groupby3.q and groupby5.q
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc".

        use class name for UDAFEvaluator in AggregationDesc when the class is not Serializable or Externalizable

        Reviewers: ashutoshc, JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D12963

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D12963?vs=40053&id=40065#toc

        BRANCH
        HIVE-5279

        ARCANIST PROJECT
        hive

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        ql/src/test/queries/clientpositive/udaf_sum_list.q
        ql/src/test/results/clientpositive/udaf_sum_list.q.out

        To: JIRA, ashutoshc, navis

        Show
        Phabricator added a comment - navis updated the revision " HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc". use class name for UDAFEvaluator in AggregationDesc when the class is not Serializable or Externalizable Reviewers: ashutoshc, JIRA REVISION DETAIL https://reviews.facebook.net/D12963 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D12963?vs=40053&id=40065#toc BRANCH HIVE-5279 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java ql/src/test/queries/clientpositive/udaf_sum_list.q ql/src/test/results/clientpositive/udaf_sum_list.q.out To: JIRA, ashutoshc, navis
        Hide
        Phabricator added a comment -

        navis updated the revision "HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc".

        Fixed test fails

        Reviewers: ashutoshc, JIRA

        REVISION DETAIL
        https://reviews.facebook.net/D12963

        CHANGE SINCE LAST DIFF
        https://reviews.facebook.net/D12963?vs=40017&id=40053#toc

        BRANCH
        HIVE-5279

        ARCANIST PROJECT
        hive

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        ql/src/test/queries/clientpositive/udaf_sum_list.q
        ql/src/test/results/clientpositive/udaf_sum_list.q.out

        To: JIRA, ashutoshc, navis

        Show
        Phabricator added a comment - navis updated the revision " HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc". Fixed test fails Reviewers: ashutoshc, JIRA REVISION DETAIL https://reviews.facebook.net/D12963 CHANGE SINCE LAST DIFF https://reviews.facebook.net/D12963?vs=40017&id=40053#toc BRANCH HIVE-5279 ARCANIST PROJECT hive AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/exec/UDAF.java ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java ql/src/test/queries/clientpositive/udaf_sum_list.q ql/src/test/results/clientpositive/udaf_sum_list.q.out To: JIRA, ashutoshc, navis
        Hide
        Navis added a comment -

        Oh, I'll check that.

        Show
        Navis added a comment - Oh, I'll check that.
        Hide
        Ashutosh Chauhan added a comment -

        Additionally following tests failed too:

        • TestContribCliDriver_udaf_example_avg.q
        • TestContribCliDriver_udaf_example_group_concat.q
        • TestContribCliDriver__udaf_example_max.q
        • TestContribCliDriver_udaf_example_max_n.q
        • TestContribCliDriver__udaf_example_min.q
        • TestContribCliDriver__udaf_example_max_n.q
        Show
        Ashutosh Chauhan added a comment - Additionally following tests failed too: TestContribCliDriver_udaf_example_avg.q TestContribCliDriver_udaf_example_group_concat.q TestContribCliDriver__udaf_example_max.q TestContribCliDriver_udaf_example_max_n.q TestContribCliDriver__udaf_example_min.q TestContribCliDriver__udaf_example_max_n.q
        Hide
        Ashutosh Chauhan added a comment -

        Following tests failed:

        • TestParse
        • TestCliDriver_autogen_colalias.q
        • TestCliDriver_create_udaf.q
        • TestCliDriver_create_view.q
        • TestCliDriver_limit_pushdown.q
        • TestCliDriver_show_functions.q
        • TestCliDriver_udaf_sum_list.q
        • TestCliDriver_udf_percentile.q
        Show
        Ashutosh Chauhan added a comment - Following tests failed: TestParse TestCliDriver_autogen_colalias.q TestCliDriver_create_udaf.q TestCliDriver_create_view.q TestCliDriver_limit_pushdown.q TestCliDriver_show_functions.q TestCliDriver_udaf_sum_list.q TestCliDriver_udf_percentile.q
        Hide
        Phabricator added a comment -

        ashutoshc has accepted the revision "HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc".

        +1

        REVISION DETAIL
        https://reviews.facebook.net/D12963

        BRANCH
        HIVE-5279

        ARCANIST PROJECT
        hive

        To: JIRA, ashutoshc, navis

        Show
        Phabricator added a comment - ashutoshc has accepted the revision " HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc". +1 REVISION DETAIL https://reviews.facebook.net/D12963 BRANCH HIVE-5279 ARCANIST PROJECT hive To: JIRA, ashutoshc, navis
        Hide
        Phabricator added a comment -

        navis requested code review of "HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc".

        Reviewers: JIRA

        HIVE-5279 Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc

        We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how previous serialization mechanism solved this but, kryo complaints that it's not Serializable and fails the query.

        The log below is the example,

        java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector
        Serialization trace:
        inputOI (org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval)
        genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc)
        aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc)
        conf (org.apache.hadoop.hive.ql.exec.GroupByOperator)
        childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator)
        childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator)
        aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork)
        at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312)
        at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261)
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256)
        at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383)
        at org.apache.h

        If this cannot be fixed in somehow, some UDAFs should be modified to be run on hive-0.13.0

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D12963

        AFFECTED FILES
        ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java
        ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java
        ql/src/test/queries/clientpositive/udaf_sum_list.q
        ql/src/test/results/clientpositive/udaf_sum_list.q.out

        MANAGE HERALD RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/30963/

        To: JIRA, navis

        Show
        Phabricator added a comment - navis requested code review of " HIVE-5279 [jira] Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc". Reviewers: JIRA HIVE-5279 Kryo cannot instantiate GenericUDAFEvaluator in GroupByDesc We didn't forced GenericUDAFEvaluator to be Serializable. I don't know how previous serialization mechanism solved this but, kryo complaints that it's not Serializable and fails the query. The log below is the example, java.lang.RuntimeException: com.esotericsoftware.kryo.KryoException: Class cannot be created (missing no-arg constructor): org.apache.hadoop.hive.serde2.objectinspector.StandardListObjectInspector Serialization trace: inputOI (org.apache.hadoop.hive.ql.udf.generic.GenericUDAFGroupOn$VersionedFloatGroupOnEval) genericUDAFEvaluator (org.apache.hadoop.hive.ql.plan.AggregationDesc) aggregators (org.apache.hadoop.hive.ql.plan.GroupByDesc) conf (org.apache.hadoop.hive.ql.exec.GroupByOperator) childOperators (org.apache.hadoop.hive.ql.exec.SelectOperator) childOperators (org.apache.hadoop.hive.ql.exec.TableScanOperator) aliasToWork (org.apache.hadoop.hive.ql.plan.MapWork) at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:312) at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:261) at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:256) at org.apache.hadoop.hive.ql.io.HiveInputFormat.pushProjectionsAndFilters(HiveInputFormat.java:383) at org.apache.h If this cannot be fixed in somehow, some UDAFs should be modified to be run on hive-0.13.0 TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D12963 AFFECTED FILES ql/src/java/org/apache/hadoop/hive/ql/plan/AggregationDesc.java ql/src/test/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSumList.java ql/src/test/queries/clientpositive/udaf_sum_list.q ql/src/test/results/clientpositive/udaf_sum_list.q.out MANAGE HERALD RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/30963/ To: JIRA, navis
        Hide
        Navis added a comment -

        The point is we've not required anything for implementation of UDAFs till now.

        Show
        Navis added a comment - The point is we've not required anything for implementation of UDAFs till now.
        Hide
        Edward Capriolo added a comment -

        I just ran into this making the collect_list udaf, you need to do things java-bean style. No-arge constructor + setter and getter. It probably was valid before but it is not now.

        Show
        Edward Capriolo added a comment - I just ran into this making the collect_list udaf, you need to do things java-bean style. No-arge constructor + setter and getter. It probably was valid before but it is not now.
        Hide
        Navis added a comment -

        Ok, sure.

        Show
        Navis added a comment - Ok, sure.
        Hide
        Ashutosh Chauhan added a comment -

        I think Kryo has other mechanisms of serialization which doesn't have that requirement. I haven't explored Kryo fully yet. Probably, we should do that in a follow-up. I think for now, we can get this one in and have this requirement in the meanwhile.
        Navis Since you have that custom udf, can you write a test case for this and submit to let Hive QA run on the patch.

        Show
        Ashutosh Chauhan added a comment - I think Kryo has other mechanisms of serialization which doesn't have that requirement. I haven't explored Kryo fully yet. Probably, we should do that in a follow-up. I think for now, we can get this one in and have this requirement in the meanwhile. Navis Since you have that custom udf, can you write a test case for this and submit to let Hive QA run on the patch.
        Hide
        Navis added a comment -

        I've applied the patch and confirmed the query working. But, reflectionUtils.newInstance() uses default constructor, which means it forces UDAFEvaluator to have it (like GenericUDAFGroupOn$VersionedFloatGroupOnEval in this case). I'm ok to make this to be a requirement of UDAFEvaluator. Other ideas?

        Show
        Navis added a comment - I've applied the patch and confirmed the query working. But, reflectionUtils.newInstance() uses default constructor, which means it forces UDAFEvaluator to have it (like GenericUDAFGroupOn$VersionedFloatGroupOnEval in this case). I'm ok to make this to be a requirement of UDAFEvaluator. Other ideas?
        Hide
        Ashutosh Chauhan added a comment - - edited

        Its a custom UDAFEvaluator GenericUDAFGroupOn which for some reason contains ObjectInspector without a no-arg constructor. Probably, none of the evaluator which we have in our tests have this. Since, evaluator is used at runtime, so it needs to be reconstructed anyways at backend. See, GroupByOperator::initializeOp(). Hence there is no good reason to serialize the constructed instance of it from frontend to backend.

        Show
        Ashutosh Chauhan added a comment - - edited Its a custom UDAFEvaluator GenericUDAFGroupOn which for some reason contains ObjectInspector without a no-arg constructor. Probably, none of the evaluator which we have in our tests have this. Since, evaluator is used at runtime, so it needs to be reconstructed anyways at backend. See, GroupByOperator::initializeOp(). Hence there is no good reason to serialize the constructed instance of it from frontend to backend.
        Hide
        Brock Noland added a comment -

        Interesting the tests didn't catch this? How did you run into it?

        Show
        Brock Noland added a comment - Interesting the tests didn't catch this? How did you run into it?
        Hide
        Ashutosh Chauhan added a comment -

        It was valid to use UDAFEvaluator without default ctor earlier on jdk6. Since, java xml serialization was lenient on that. However, on jdk7 you are required to provide a no-arg ctor even for java xml serialization. So, even without Kryo, you would run into this problem, once Hive moves to jdk7.

        However, patch I have uploaded eliminates this requirement altogether, since genericUDAFEvaluator (user provided class) is no longer stored as field in AggregatorDesc. So, if this patch works it won't matter if you have default ctor or not either for Kryo or jdk6 or jdk7.

        Show
        Ashutosh Chauhan added a comment - It was valid to use UDAFEvaluator without default ctor earlier on jdk6. Since, java xml serialization was lenient on that. However, on jdk7 you are required to provide a no-arg ctor even for java xml serialization. So, even without Kryo, you would run into this problem, once Hive moves to jdk7. However, patch I have uploaded eliminates this requirement altogether, since genericUDAFEvaluator (user provided class) is no longer stored as field in AggregatorDesc. So, if this patch works it won't matter if you have default ctor or not either for Kryo or jdk6 or jdk7.
        Hide
        Ashutosh Chauhan added a comment -

        Here is an illustrative patch which should fix the issue. Navis Can you give it a try?

        Show
        Ashutosh Chauhan added a comment - Here is an illustrative patch which should fix the issue. Navis Can you give it a try?
        Hide
        Navis added a comment -

        I considered that. But the immediate question is, was it valid to use UDAFEvaluator without default constructor before? I should check that, in tomorrow.

        Show
        Navis added a comment - I considered that. But the immediate question is, was it valid to use UDAFEvaluator without default constructor before? I should check that, in tomorrow.
        Hide
        Ashutosh Chauhan added a comment -

        I think this can be fixed by making following changes in AggregationDesc class:
        1. Mark genericUDAFEvaluator as transient.
        2. Store the class name of GenericUDAFEvaluator as string.
        3. In getGenericUDAFEvaluator(), using reflection on className string, instantiate GenericUDAFEvaluator and return it.

        Since none of the callers of getGenericUDAFEvaluator make any use of state of genericUDAFEvaluator, this should work. Can you give it a try?

        Show
        Ashutosh Chauhan added a comment - I think this can be fixed by making following changes in AggregationDesc class: 1. Mark genericUDAFEvaluator as transient. 2. Store the class name of GenericUDAFEvaluator as string. 3. In getGenericUDAFEvaluator(), using reflection on className string, instantiate GenericUDAFEvaluator and return it. Since none of the callers of getGenericUDAFEvaluator make any use of state of genericUDAFEvaluator, this should work. Can you give it a try?

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development