Hive
  1. Hive
  2. HIVE-2332

If all of the parameters of distinct functions are exists in group by columns, query fails in runtime

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 0.11.0
    • Component/s: Query Processor
    • Labels:
      None

      Description

      select sum(key_int1), sum(distinct key_int1) from t1 group by key_int1;

      fails with message..

      FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask
      

      hadoop says..

      Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1
      	at java.util.ArrayList.RangeCheck(ArrayList.java:547)
      	at java.util.ArrayList.get(ArrayList.java:322)
      	at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:95)
      	at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:86)
      	at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:252)
      	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:188)
      	at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:197)
      	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
      	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
      	at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:85)
      	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471)
      	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744)
      	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:532)
      

      I think the deficient number of key expression, compared to number of key column, is the problem, which should be equal or more.
      Would it be solved if add some key expression? I'll try.

      1. ASF.LICENSE.NOT.GRANTED--HIVE-2332.D663.1.patch
        11 kB
        Phabricator
      2. HIVE-2332.1.patch.txt
        3 kB
        Navis
      3. HIVE-2332.2.patch.txt
        11 kB
        Navis

        Activity

        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #79 (See https://builds.apache.org/job/Hive-trunk-hadoop2/79/)
        HIVE-3920 Change test for HIVE-2332
        (Ashutosh Chauhan and Navis via namit) (Revision 1436199)

        Result = FAILURE
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436199
        Files :

        • /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q
        • /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #79 (See https://builds.apache.org/job/Hive-trunk-hadoop2/79/ ) HIVE-3920 Change test for HIVE-2332 (Ashutosh Chauhan and Navis via namit) (Revision 1436199) Result = FAILURE namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436199 Files : /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1928 (See https://builds.apache.org/job/Hive-trunk-h0.21/1928/)
        HIVE-3920 Change test for HIVE-2332
        (Ashutosh Chauhan and Navis via namit) (Revision 1436199)

        Result = SUCCESS
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436199
        Files :

        • /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q
        • /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1928 (See https://builds.apache.org/job/Hive-trunk-h0.21/1928/ ) HIVE-3920 Change test for HIVE-2332 (Ashutosh Chauhan and Navis via namit) (Revision 1436199) Result = SUCCESS namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436199 Files : /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Hide
        Hudson added a comment -

        Integrated in hive-trunk-hadoop1 #33 (See https://builds.apache.org/job/hive-trunk-hadoop1/33/)
        HIVE-3920 Change test for HIVE-2332
        (Ashutosh Chauhan and Navis via namit) (Revision 1436199)

        Result = ABORTED
        namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436199
        Files :

        • /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q
        • /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Show
        Hudson added a comment - Integrated in hive-trunk-hadoop1 #33 (See https://builds.apache.org/job/hive-trunk-hadoop1/33/ ) HIVE-3920 Change test for HIVE-2332 (Ashutosh Chauhan and Navis via namit) (Revision 1436199) Result = ABORTED namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1436199 Files : /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-hadoop2 #77 (See https://builds.apache.org/job/Hive-trunk-hadoop2/77/)
        HIVE-2332 : If all of the parameters of distinct functions are exists in group by columns, query fails in runtime (Navis via Ashutosh Chauhan) (Revision 1435790)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435790
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
        • /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q
        • /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-hadoop2 #77 (See https://builds.apache.org/job/Hive-trunk-hadoop2/77/ ) HIVE-2332 : If all of the parameters of distinct functions are exists in group by columns, query fails in runtime (Navis via Ashutosh Chauhan) (Revision 1435790) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435790 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Hide
        Hudson added a comment -

        Integrated in Hive-trunk-h0.21 #1926 (See https://builds.apache.org/job/Hive-trunk-h0.21/1926/)
        HIVE-2332 : If all of the parameters of distinct functions are exists in group by columns, query fails in runtime (Navis via Ashutosh Chauhan) (Revision 1435790)

        Result = FAILURE
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435790
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
        • /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q
        • /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Show
        Hudson added a comment - Integrated in Hive-trunk-h0.21 #1926 (See https://builds.apache.org/job/Hive-trunk-h0.21/1926/ ) HIVE-2332 : If all of the parameters of distinct functions are exists in group by columns, query fails in runtime (Navis via Ashutosh Chauhan) (Revision 1435790) Result = FAILURE hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435790 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Hide
        Hudson added a comment -

        Integrated in hive-trunk-hadoop1 #31 (See https://builds.apache.org/job/hive-trunk-hadoop1/31/)
        HIVE-2332 : If all of the parameters of distinct functions are exists in group by columns, query fails in runtime (Navis via Ashutosh Chauhan) (Revision 1435790)

        Result = ABORTED
        hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435790
        Files :

        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
        • /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java
        • /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q
        • /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Show
        Hudson added a comment - Integrated in hive-trunk-hadoop1 #31 (See https://builds.apache.org/job/hive-trunk-hadoop1/31/ ) HIVE-2332 : If all of the parameters of distinct functions are exists in group by columns, query fails in runtime (Navis via Ashutosh Chauhan) (Revision 1435790) Result = ABORTED hashutosh : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1435790 Files : /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java /hive/trunk/ql/src/test/queries/clientpositive/groupby_distinct_samekey.q /hive/trunk/ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        Hide
        Ashutosh Chauhan added a comment -

        Committed to trunk. Thanks, Navis!

        Show
        Ashutosh Chauhan added a comment - Committed to trunk. Thanks, Navis!
        Hide
        Ashutosh Chauhan added a comment -

        Verified that bug still exists in trunk. Patch looks good. +1 Will commit if tests pass.

        Show
        Ashutosh Chauhan added a comment - Verified that bug still exists in trunk. Patch looks good. +1 Will commit if tests pass.
        Hide
        Ashutosh Chauhan added a comment -

        Unlinking from 0.9

        Show
        Ashutosh Chauhan added a comment - Unlinking from 0.9
        Hide
        Ashutosh Chauhan added a comment -

        Oh, I missed it. Marking Patch Available then.

        Show
        Ashutosh Chauhan added a comment - Oh, I missed it. Marking Patch Available then.
        Hide
        Navis added a comment -

        It's already on phabricator by Namit.

        Show
        Navis added a comment - It's already on phabricator by Namit.
        Hide
        Ashutosh Chauhan added a comment -

        Canceling patch as it first needs to address Namit's comments.

        Show
        Ashutosh Chauhan added a comment - Canceling patch as it first needs to address Namit's comments.
        Hide
        Namit Jain added a comment -

        Navis, please use the following instructions in future for patches
        https://cwiki.apache.org/Hive/phabricatorcodereview.html.

        We are planning to move to phabricator from review board.

        I have already done this for this jira.
        Will review it soon.

        Show
        Namit Jain added a comment - Navis, please use the following instructions in future for patches https://cwiki.apache.org/Hive/phabricatorcodereview.html . We are planning to move to phabricator from review board. I have already done this for this jira. Will review it soon.
        Hide
        Phabricator added a comment -

        njain requested code review of "HIVE-2332 [jira] If all of the parameters of distinct functions are exists in group by columns, query fails in runtime".
        Reviewers: JIRA

        HIVE-2332

        select sum(key_int1), sum(distinct key_int1) from t1 group by key_int1;

        fails with message.. <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">FAILED: Execution Error, <span class="code-keyword">return</span> code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask</pre> </div></div>

        hadoop says.. <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:95) at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:86) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:252) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:188) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.e
        xec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:85) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:532)</pre> </div></div>

        I think the deficient number of key expression, compared to number of key column, is the problem, which should be equal or more.
        Would it be solved if add some key expression? I'll try.

        TEST PLAN
        EMPTY

        REVISION DETAIL
        https://reviews.facebook.net/D663

        AFFECTED FILES
        ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out
        ql/src/test/queries/clientpositive/groupby_distinct_samekey.q
        ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java
        ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java

        MANAGE HERALD DIFFERENTIAL RULES
        https://reviews.facebook.net/herald/view/differential/

        WHY DID I GET THIS EMAIL?
        https://reviews.facebook.net/herald/transcript/1473/

        Tip: use the X-Herald-Rules header to filter Herald messages in your client.

        Show
        Phabricator added a comment - njain requested code review of " HIVE-2332 [jira] If all of the parameters of distinct functions are exists in group by columns, query fails in runtime". Reviewers: JIRA HIVE-2332 select sum(key_int1), sum(distinct key_int1) from t1 group by key_int1; fails with message.. <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">FAILED: Execution Error, <span class="code-keyword">return</span> code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask</pre> </div></div> hadoop says.. <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at java.util.ArrayList.RangeCheck(ArrayList.java:547) at java.util.ArrayList.get(ArrayList.java:322) at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.init(StandardStructObjectInspector.java:95) at org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector.(StandardStructObjectInspector.java:86) at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory.getStandardStructObjectInspector(ObjectInspectorFactory.java:252) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.initEvaluatorsAndReturnStruct(ReduceSinkOperator.java:188) at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.processOp(ReduceSinkOperator.java:197) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.e xec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:85) at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:471) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:744) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:532)</pre> </div></div> I think the deficient number of key expression, compared to number of key column, is the problem, which should be equal or more. Would it be solved if add some key expression? I'll try. TEST PLAN EMPTY REVISION DETAIL https://reviews.facebook.net/D663 AFFECTED FILES ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out ql/src/test/queries/clientpositive/groupby_distinct_samekey.q ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java MANAGE HERALD DIFFERENTIAL RULES https://reviews.facebook.net/herald/view/differential/ WHY DID I GET THIS EMAIL? https://reviews.facebook.net/herald/transcript/1473/ Tip: use the X-Herald-Rules header to filter Herald messages in your client.
        Hide
        Navis added a comment -
        Show
        Navis added a comment - https://reviews.apache.org/r/1314/
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1314/
        -----------------------------------------------------------

        (Updated 2011-12-07 07:01:30.547976)

        Review request for hive, John Sichi and Carl Steinbach.

        Changes
        -------

        Adding null keys induced confusions especially for optimizers. This patch just modifies key-order minimizing side effects.

        Summary
        -------

        If all of the distinct params are in group by keys, union column reserved for distinct params would not be added, which making problems initializing RS operator.

        This patch is just a simple bypass adding dummy expression for the union column. Someone would know better way to resolve the problem.

        This addresses bug HIVE-2332.
        https://issues.apache.org/jira/browse/HIVE-2332

        Diffs (updated)


        ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 732a5aa
        ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java c6ae55d
        ql/src/test/queries/clientpositive/groupby_distinct_samekey.q PRE-CREATION
        ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/1314/diff

        Testing
        -------

        added clientpositive/groupby_distinct_samekey.q

        Thanks,

        Navis

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1314/ ----------------------------------------------------------- (Updated 2011-12-07 07:01:30.547976) Review request for hive, John Sichi and Carl Steinbach. Changes ------- Adding null keys induced confusions especially for optimizers. This patch just modifies key-order minimizing side effects. Summary ------- If all of the distinct params are in group by keys, union column reserved for distinct params would not be added, which making problems initializing RS operator. This patch is just a simple bypass adding dummy expression for the union column. Someone would know better way to resolve the problem. This addresses bug HIVE-2332 . https://issues.apache.org/jira/browse/HIVE-2332 Diffs (updated) ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 732a5aa ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java c6ae55d ql/src/test/queries/clientpositive/groupby_distinct_samekey.q PRE-CREATION ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1314/diff Testing ------- added clientpositive/groupby_distinct_samekey.q Thanks, Navis
        Hide
        Carl Steinbach added a comment -

        Is this patch ready for review? Is so, please change the status to Patch Available. Thanks.

        Show
        Carl Steinbach added a comment - Is this patch ready for review? Is so, please change the status to Patch Available. Thanks.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1314/
        -----------------------------------------------------------

        (Updated 2011-11-15 08:56:07.599798)

        Review request for hive and Ashutosh Chauhan.

        Changes
        -------

        Rebased to trunk.

        Summary
        -------

        If all of the distinct params are in group by keys, union column reserved for distinct params would not be added, which making problems initializing RS operator.

        This patch is just a simple bypass adding dummy expression for the union column. Someone would know better way to resolve the problem.

        This addresses bug HIVE-2332.
        https://issues.apache.org/jira/browse/HIVE-2332

        Diffs (updated)


        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d60ba09
        ql/src/test/queries/clientpositive/groupby_distinct_samekey.q PRE-CREATION
        ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/1314/diff

        Testing (updated)
        -------

        added clientpositive/groupby_distinct_samekey.q

        Thanks,

        Navis

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1314/ ----------------------------------------------------------- (Updated 2011-11-15 08:56:07.599798) Review request for hive and Ashutosh Chauhan. Changes ------- Rebased to trunk. Summary ------- If all of the distinct params are in group by keys, union column reserved for distinct params would not be added, which making problems initializing RS operator. This patch is just a simple bypass adding dummy expression for the union column. Someone would know better way to resolve the problem. This addresses bug HIVE-2332 . https://issues.apache.org/jira/browse/HIVE-2332 Diffs (updated) ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d60ba09 ql/src/test/queries/clientpositive/groupby_distinct_samekey.q PRE-CREATION ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1314/diff Testing (updated) ------- added clientpositive/groupby_distinct_samekey.q Thanks, Navis
        Hide
        Ashutosh Chauhan added a comment -

        Assigning to Navis. Navis, also mark the issue as Patch Available if its ready for review.

        Show
        Ashutosh Chauhan added a comment - Assigning to Navis. Navis, also mark the issue as Patch Available if its ready for review.
        Hide
        jiraposter@reviews.apache.org added a comment -

        -----------------------------------------------------------
        This is an automatically generated e-mail. To reply, visit:
        https://reviews.apache.org/r/1314/
        -----------------------------------------------------------

        Review request for hive.

        Summary
        -------

        If all of the distinct params are in group by keys, union column reserved for distinct params would not be added, which making problems initializing RS operator.

        This patch is just a simple bypass adding dummy expression for the union column. Someone would know better way to resolve the problem.

        This addresses bug HIVE-2332.
        https://issues.apache.org/jira/browse/HIVE-2332

        Diffs


        ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java dc9ab1b
        ql/src/test/queries/clientpositive/groupby_distinct_samekey.q PRE-CREATION
        ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out PRE-CREATION

        Diff: https://reviews.apache.org/r/1314/diff

        Testing
        -------

        Thanks,

        Navis

        Show
        jiraposter@reviews.apache.org added a comment - ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1314/ ----------------------------------------------------------- Review request for hive. Summary ------- If all of the distinct params are in group by keys, union column reserved for distinct params would not be added, which making problems initializing RS operator. This patch is just a simple bypass adding dummy expression for the union column. Someone would know better way to resolve the problem. This addresses bug HIVE-2332 . https://issues.apache.org/jira/browse/HIVE-2332 Diffs ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java dc9ab1b ql/src/test/queries/clientpositive/groupby_distinct_samekey.q PRE-CREATION ql/src/test/results/clientpositive/groupby_distinct_samekey.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1314/diff Testing ------- Thanks, Navis
        Hide
        Navis added a comment -

        add test case

        Show
        Navis added a comment - add test case
        Hide
        Navis added a comment -

        1. Add deficient columns lastly to avoid index mismatch with RR of HASH GBY
        2. ExprNodeNullDesc to ExprNodeFieldDesc to avoid NPE

        Show
        Navis added a comment - 1. Add deficient columns lastly to avoid index mismatch with RR of HASH GBY 2. ExprNodeNullDesc to ExprNodeFieldDesc to avoid NPE
        Hide
        John Sichi added a comment -

        Once you have the patch ready, also please create a Review Board entry:

        https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-ReviewProcess

        Thanks!

        Show
        John Sichi added a comment - Once you have the patch ready, also please create a Review Board entry: https://cwiki.apache.org/confluence/display/Hive/HowToContribute#HowToContribute-ReviewProcess Thanks!
        Hide
        John Sichi added a comment -

        Good catch. Can you submit an updated patch which includes the test case (.q and .q.out)?

        Show
        John Sichi added a comment - Good catch. Can you submit an updated patch which includes the test case (.q and .q.out)?
        Hide
        Navis added a comment -

        Simple bypass.

        Removing not-used output key column made problems in following operations(GBY). So I've added dummy ExprNodeNullDesc for key expression and query has succeeded.

        Show
        Navis added a comment - Simple bypass. Removing not-used output key column made problems in following operations(GBY). So I've added dummy ExprNodeNullDesc for key expression and query has succeeded.

          People

          • Assignee:
            Navis
            Reporter:
            Navis
          • Votes:
            1 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development