Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1869

Incorrect result when sorting table with small files

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 0.9.0, 0.10.0, 0.10.1
    • Fix Version/s: 0.11.0, 0.12.0
    • Component/s: Data Shuffle
    • Labels:
      None

      Description

      A shuffle data can read directly when data is located in same host.
      If a chunk is out of scope in cardinality, it must be removed
      A remote chunk returns NO_CONTENT but local chunk does not ignore the data

        Activity

        Hide
        githubbot ASF GitHub Bot added a comment -

        GitHub user jinossy opened a pull request:

        https://github.com/apache/tajo/pull/769

        TAJO-1869: Incorrect result when sorting table with small files

        You can merge this pull request into a Git repository by running:

        $ git pull https://github.com/jinossy/tajo TAJO-1869

        Alternatively you can review and apply these changes as the patch at:

        https://github.com/apache/tajo/pull/769.patch

        To close this pull request, make a commit to your master/trunk branch
        with (at least) the following in the commit message:

        This closes #769


        commit b72fb61a0ba74f9fd6b8fd410b72eb27b756f979
        Author: Jinho Kim <jhkim@apache.org>
        Date: 2015-09-18T05:12:15Z

        TAJO-1869: Incorrect result when sorting table with small files


        Show
        githubbot ASF GitHub Bot added a comment - GitHub user jinossy opened a pull request: https://github.com/apache/tajo/pull/769 TAJO-1869 : Incorrect result when sorting table with small files You can merge this pull request into a Git repository by running: $ git pull https://github.com/jinossy/tajo TAJO-1869 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/769.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #769 commit b72fb61a0ba74f9fd6b8fd410b72eb27b756f979 Author: Jinho Kim <jhkim@apache.org> Date: 2015-09-18T05:12:15Z TAJO-1869 : Incorrect result when sorting table with small files
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user jihoonson commented on the pull request:

        https://github.com/apache/tajo/pull/769#issuecomment-141643782

        +1 LGTM!

        Show
        githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/769#issuecomment-141643782 +1 LGTM!
        Hide
        githubbot ASF GitHub Bot added a comment -

        Github user asfgit closed the pull request at:

        https://github.com/apache/tajo/pull/769

        Show
        githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/769
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-master-CODEGEN-build #524 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/524/)
        TAJO-1869: Incorrect result when sorting table with small files. (jhkim: rev 8f00bf7433e7dd99d843e9c97894bdabd81ccb50)

        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table2.tbl
        • tajo-core-tests/src/test/resources/queries/TestSortQuery/create_table_with_unique_small_dataset.sql
        • tajo-core-tests/src/test/resources/results/TestSortQuery/testOutOfScope.result
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
        • tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table4.tbl
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table3.tbl
        • tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
        • tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/RpcClientManager.java
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table5.tbl
        • CHANGES
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table1.tbl
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
        • tajo-core-tests/src/test/resources/queries/TestSortQuery/testOutOfScope.sql
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #524 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/524/ ) TAJO-1869 : Incorrect result when sorting table with small files. (jhkim: rev 8f00bf7433e7dd99d843e9c97894bdabd81ccb50) tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table2.tbl tajo-core-tests/src/test/resources/queries/TestSortQuery/create_table_with_unique_small_dataset.sql tajo-core-tests/src/test/resources/results/TestSortQuery/testOutOfScope.result tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table4.tbl tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table3.tbl tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/RpcClientManager.java tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table5.tbl CHANGES tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table1.tbl tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-core-tests/src/test/resources/queries/TestSortQuery/testOutOfScope.sql
        Hide
        jhkim Jinho Kim added a comment -

        committed it
        Thanks.

        Show
        jhkim Jinho Kim added a comment - committed it Thanks.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-master-build #884 (See https://builds.apache.org/job/Tajo-master-build/884/)
        TAJO-1869: Incorrect result when sorting table with small files. (jhkim: rev 8f00bf7433e7dd99d843e9c97894bdabd81ccb50)

        • tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table2.tbl
        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
        • tajo-core-tests/src/test/resources/results/TestSortQuery/testOutOfScope.result
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table5.tbl
        • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
        • tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/RpcClientManager.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
        • CHANGES
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table3.tbl
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table4.tbl
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table1.tbl
        • tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core-tests/src/test/resources/queries/TestSortQuery/create_table_with_unique_small_dataset.sql
        • tajo-core-tests/src/test/resources/queries/TestSortQuery/testOutOfScope.sql
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #884 (See https://builds.apache.org/job/Tajo-master-build/884/ ) TAJO-1869 : Incorrect result when sorting table with small files. (jhkim: rev 8f00bf7433e7dd99d843e9c97894bdabd81ccb50) tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table2.tbl tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-core-tests/src/test/resources/results/TestSortQuery/testOutOfScope.result tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table5.tbl tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/RpcClientManager.java tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java CHANGES tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table3.tbl tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table4.tbl tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table1.tbl tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core-tests/src/test/resources/queries/TestSortQuery/create_table_with_unique_small_dataset.sql tajo-core-tests/src/test/resources/queries/TestSortQuery/testOutOfScope.sql
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Tajo-0.11.0-build #60 (See https://builds.apache.org/job/Tajo-0.11.0-build/60/)
        TAJO-1869: Incorrect result when sorting table with small files. (jhkim: rev e5c6450e589cbe7dec274e9bae93d87e404a99cc)

        • tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table4.tbl
        • tajo-core-tests/src/test/resources/results/TestSortQuery/testOutOfScope.result
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
        • tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core-tests/src/test/resources/queries/TestSortQuery/create_table_with_unique_small_dataset.sql
        • tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java
        • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
        • tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/RpcClientManager.java
        • CHANGES
        • tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table5.tbl
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table3.tbl
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table2.tbl
        • tajo-core-tests/src/test/resources/queries/TestSortQuery/testOutOfScope.sql
        • tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java
        • tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table1.tbl
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Tajo-0.11.0-build #60 (See https://builds.apache.org/job/Tajo-0.11.0-build/60/ ) TAJO-1869 : Incorrect result when sorting table with small files. (jhkim: rev e5c6450e589cbe7dec274e9bae93d87e404a99cc) tajo-core-tests/src/test/java/org/apache/tajo/engine/query/TestSortQuery.java tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table4.tbl tajo-core-tests/src/test/resources/results/TestSortQuery/testOutOfScope.result tajo-core/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core-tests/src/test/resources/queries/TestSortQuery/create_table_with_unique_small_dataset.sql tajo-core/src/main/java/org/apache/tajo/engine/planner/physical/ExternalSortExec.java tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-rpc/tajo-rpc-protobuf/src/main/java/org/apache/tajo/rpc/RpcClientManager.java CHANGES tajo-core/src/main/java/org/apache/tajo/worker/Fetcher.java tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table5.tbl tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table3.tbl tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table2.tbl tajo-core-tests/src/test/resources/queries/TestSortQuery/testOutOfScope.sql tajo-core/src/main/java/org/apache/tajo/worker/TaskImpl.java tajo-core-tests/src/test/resources/dataset/TestSortQuery/table3/table1.tbl

          People

          • Assignee:
            jhkim Jinho Kim
            Reporter:
            jhkim Jinho Kim
          • Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development