Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-729

PreLogicalPlanVerifier verifies distinct aggregation functions incorrectly.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0, 0.9.0
    • Component/s: Planner/Optimizer
    • Labels:
      None

      Description

      Since the current Tajo cannot process distinct aggregation functions for multiple columns, PreLogicalPlanVerifier prevents such a case. But, PreLogicalPlanVerifier checks it incorrectly. As a result, it causes a verification error even though a query can be actually executed in Tajo.

      See the below example. This query can be actually executed in Tajo.

      select
        count(distinct l_orderkey) as v0,
        sum(l_orderkey) as v1,
        sum(l_linenumber) as v2,
        count(*) as v4
      from
        lineitem
      group by
        l_orderkey;
      

      But, it causes the following verification error.

      org.apache.tajo.engine.planner.PlanningException: different DISTINCT columns are not supported yet: l_orderkey, l_linenumber
        at org.apache.tajo.engine.planner.PreLogicalPlanVerifier.visitProjection(PreLogicalPlanVerifier.java:89)
        at org.apache.tajo.engine.planner.PreLogicalPlanVerifier.visitProjection(PreLogicalPlanVerifier.java:33)
        at org.apache.tajo.engine.planner.BaseAlgebraVisitor.visit(BaseAlgebraVisitor.java:55)
        at org.apache.tajo.engine.planner.PreLogicalPlanVerifier.verify(PreLogicalPlanVerifier.java:52)
        at org.apache.tajo.master.GlobalEngine.createLogicalPlan(GlobalEngine.java:247)
        at org.apache.tajo.master.GlobalEngine.executeQuery(GlobalEngine.java:138)
        at org.apache.tajo.master.TajoMasterClientService$TajoMasterClientProtocolServiceHandler.submitQuery(TajoMasterClientService.java:286)
        at org.apache.tajo.ipc.TajoMasterClientProtocol$TajoMasterClientProtocolService$2.callBlockingMethod(TajoMasterClientProtocol.java:513)
        at org.apache.tajo.rpc.BlockingRpcServer$ServerHandler.messageReceived(BlockingRpcServer.java:103)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
        at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:109)
        at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:312)
        at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:90)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)
      
      1. TAJO-729.patch
        3 kB
        Hyunsik Choi

        Activity

        Hide
        hyunsik Hyunsik Choi added a comment -

        Created a review request against branch master in reviewboard
        https://reviews.apache.org/r/19841/

        Show
        hyunsik Hyunsik Choi added a comment - Created a review request against branch master in reviewboard https://reviews.apache.org/r/19841/
        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user hyunsik opened a pull request:

        https://github.com/apache/tajo/pull/2

        TAJO-729: PreLogicalPlanVerifier verifies distinct aggregation functions...

        ... incorrectly. (hyunsik)

        https://reviews.apache.org/r/19841/

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/hyunsik/tajo TAJO-729

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/2.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #2


        commit 644c83ee53a059f62ac525945dfa883d6197ec0e
        Author: Hyunsik Choi <hyunsik@apache.org>
        Date: 2014-03-31T13:19:14Z

        TAJO-729: PreLogicalPlanVerifier verifies distinct aggregation functions incorrectly. (hyunsik)


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user hyunsik opened a pull request: https://github.com/apache/tajo/pull/2 TAJO-729 : PreLogicalPlanVerifier verifies distinct aggregation functions... ... incorrectly. (hyunsik) https://reviews.apache.org/r/19841/ You can merge this pull request into a Git repository by running: $ git pull https://github.com/hyunsik/tajo TAJO-729 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/2.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2 commit 644c83ee53a059f62ac525945dfa883d6197ec0e Author: Hyunsik Choi <hyunsik@apache.org> Date: 2014-03-31T13:19:14Z TAJO-729 : PreLogicalPlanVerifier verifies distinct aggregation functions incorrectly. (hyunsik)
        Hide
        tajoqa Tajo QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12637820/TAJO-729.patch
        against master revision 40851e5.

        +1 @author. The patch does not contain any @author tags.

        +1 tests included. The patch appears to include 3 new or modified test files.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The applied patch does not increase the total number of javadoc warnings.

        +1 checkstyle. The patch generated 0 code style errors.

        -1 findbugs. The patch appears to introduce 197 new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend.

        Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/289//testReport/
        Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/289//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html
        Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/289//console

        This message is automatically generated.

        Show
        tajoqa Tajo QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12637820/TAJO-729.patch against master revision 40851e5. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The applied patch does not increase the total number of javadoc warnings. +1 checkstyle. The patch generated 0 code style errors. -1 findbugs. The patch appears to introduce 197 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in tajo-core/tajo-core-backend. Test results: https://builds.apache.org/job/PreCommit-TAJO-Build/289//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-TAJO-Build/289//artifact/incubator-tajo/patchprocess/newPatchFindbugsWarningstajo-core-backend.html Console output: https://builds.apache.org/job/PreCommit-TAJO-Build/289//console This message is automatically generated.
        Hide
        hyunsik Hyunsik Choi added a comment -

        committed it to master and branch-0.8.0. thanks for the review.

        Show
        hyunsik Hyunsik Choi added a comment - committed it to master and branch-0.8.0. thanks for the review.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #156 (See https://builds.apache.org/job/Tajo-master-build/156/)
        TAJO-729: PreLogicalPlanVerifier verifies distinct aggregation functions incorrectly. (hyunsik) (hyunsik: rev bbee120d9d0bf7ead68b9fb49a674e1841718cc9)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java
        • tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testDistinctAggregation6.result
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testDistinctAggregation6.sql
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #156 (See https://builds.apache.org/job/Tajo-master-build/156/ ) TAJO-729 : PreLogicalPlanVerifier verifies distinct aggregation functions incorrectly. (hyunsik) (hyunsik: rev bbee120d9d0bf7ead68b9fb49a674e1841718cc9) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testDistinctAggregation6.result CHANGES.txt tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testDistinctAggregation6.sql tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.8.0-build #57 (See https://builds.apache.org/job/Tajo-0.8.0-build/57/)
        TAJO-729: PreLogicalPlanVerifier verifies distinct aggregation functions incorrectly. (hyunsik) (hyunsik: rev e5519cf0badfa997f623fe1137f5e234edfecc66)

        • tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testDistinctAggregation6.result
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testDistinctAggregation6.sql
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.8.0-build #57 (See https://builds.apache.org/job/Tajo-0.8.0-build/57/ ) TAJO-729 : PreLogicalPlanVerifier verifies distinct aggregation functions incorrectly. (hyunsik) (hyunsik: rev e5519cf0badfa997f623fe1137f5e234edfecc66) tajo-core/tajo-core-backend/src/test/resources/results/TestGroupByQuery/testDistinctAggregation6.result tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/query/TestGroupByQuery.java CHANGES.txt tajo-core/tajo-core-backend/src/test/resources/queries/TestGroupByQuery/testDistinctAggregation6.sql tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PreLogicalPlanVerifier.java
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik commented on the pull request:

        https://github.com/apache/tajo/pull/2#issuecomment-39645030

        committed

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/2#issuecomment-39645030 committed
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user hyunsik closed the pull request at:

        https://github.com/apache/tajo/pull/2

        Show
        githubbot ASF GitHub Bot added a comment - Github user hyunsik closed the pull request at: https://github.com/apache/tajo/pull/2

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development