Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-464

Rename the name 'partition', actually meaning shuffle to 'shuffle'.

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: None
    • Labels:
      None

      Description

      So far, we have used the word 'partition' to indicate shuffle. This name was originated from repartition in the database studies. However, this name makes us hard to distinguish the shuffle and table partition. We ned to change such names and prefix into 'shuffle'.

      1. TAJO-464_2014-01-01_22:59:40.patch
        97 kB
        Hyunsik Choi
      2. TAJO-464_20140102_13:46:34.patch
        99 kB
        Hyunsik Choi
      3. TAJO-464.patch
        97 kB
        Hyunsik Choi

        Activity

        Hide
        coderplay Min Zhou added a comment -

        I wasn't confused with that However, totally agree that distinguish those 2 concepts.

        Show
        coderplay Min Zhou added a comment - I wasn't confused with that However, totally agree that distinguish those 2 concepts.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Created a review request in reviewboard https://reviews.apache.org/r/16542/
        against branch asf/master

        Show
        hyunsik Hyunsik Choi added a comment - Created a review request in reviewboard https://reviews.apache.org/r/16542/ against branch asf/master
        Hide
        charsyam DaeMyung Kang added a comment -

        This patch looks good to me.

        However, in tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java
        just remove line 40 "this.tableName = tableName;", it is already assigned by super.

        Show
        charsyam DaeMyung Kang added a comment - This patch looks good to me. However, in tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java just remove line 40 "this.tableName = tableName;", it is already assigned by super.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Updated the review request against branch master in reviewboard
        https://reviews.apache.org/r/16542/

        Show
        hyunsik Hyunsik Choi added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/16542/
        Hide
        hyunsik Hyunsik Choi added a comment - - edited

        DaeMyung Kang Thank you for the review. I've just uploaded the updated patch that reflects your comment.

        Show
        hyunsik Hyunsik Choi added a comment - - edited DaeMyung Kang Thank you for the review. I've just uploaded the updated patch that reflects your comment.
        Hide
        jihoonson Jihoon Son added a comment -

        Thanks, Hyunsik. This is a definitely necessary work.
        However, I'm wondering that 'shuffle' is the proper name in our query execution. As you know, the name of 'shuffle' comes from a phase of Hadoop/MapReduce that includes several processes such as sort and data repartition.
        Meanwhile, in our query execution, the intermediate data are just repartitioned between the execution blocks.
        So, in my opinion, 'repartition' is the more proper name than 'shuffle'.

        I left some more comments at RB.

        Show
        jihoonson Jihoon Son added a comment - Thanks, Hyunsik. This is a definitely necessary work. However, I'm wondering that 'shuffle' is the proper name in our query execution. As you know, the name of 'shuffle' comes from a phase of Hadoop/MapReduce that includes several processes such as sort and data repartition. Meanwhile, in our query execution, the intermediate data are just repartitioned between the execution blocks. So, in my opinion, 'repartition' is the more proper name than 'shuffle'. I left some more comments at RB.
        Hide
        hyunsik Hyunsik Choi added a comment -

        Updated the review request against branch master in reviewboard
        https://reviews.apache.org/r/16542/

        Show
        hyunsik Hyunsik Choi added a comment - Updated the review request against branch master in reviewboard https://reviews.apache.org/r/16542/
        Hide
        hyunsik Hyunsik Choi added a comment -

        committed it to master. Thank you for the reviews.

        Show
        hyunsik Hyunsik Choi added a comment - committed it to master. Thank you for the reviews.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Tajo-trunk-postcommit #661 (See https://builds.apache.org/job/Tajo-trunk-postcommit/661/)
        TAJO-464: Rename the name 'partition', actually meaning shuffle to 'shuffle'. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=bb7e6b6b3ca3f729e292df3b5905a46fe773f392)

        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/Partitioner.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
        • tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/StoreTableExec.java
        • CHANGES.txt
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashPartitioner.java
        • tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ShuffleFileWriteNode.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/global/TestMasterPlan.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PartitionedStoreExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashShuffleFileWriteExec.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java
        • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestRepartitioner.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PersistentStoreNode.java
        • tajo-core/tajo-core-backend/src/main/resources/webapps/worker/queryunit.jsp
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java
        • tajo-core/tajo-core-backend/src/main/resources/webapps/worker/queryplan.jsp
        • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/IndexedStoreExec.java
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Tajo-trunk-postcommit #661 (See https://builds.apache.org/job/Tajo-trunk-postcommit/661/ ) TAJO-464 : Rename the name 'partition', actually meaning shuffle to 'shuffle'. (hyunsik) (hyunsik: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=bb7e6b6b3ca3f729e292df3b5905a46fe773f392 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/SubQuery.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/Task.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/Partitioner.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java tajo-core/tajo-core-pullserver/src/main/java/org/apache/tajo/pullserver/TajoPullServerService.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/StoreTableNode.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/StoreTableExec.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashPartitioner.java tajo-core/tajo-core-backend/src/main/proto/TajoWorkerProtocol.proto tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/ShuffleFileWriteNode.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/global/TestMasterPlan.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/RangeShuffleFileWriteExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PartitionedStoreExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/HashShuffleFileWriteExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnit.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/Repartitioner.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/logical/PersistentStoreNode.java tajo-core/tajo-core-backend/src/main/resources/webapps/worker/queryunit.jsp tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/master/querymaster/QueryUnitAttempt.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/MasterPlan.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/worker/TaskAttemptContext.java tajo-core/tajo-core-backend/src/main/resources/webapps/worker/queryplan.jsp tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/IndexedStoreExec.java

          People

          • Assignee:
            hyunsik Hyunsik Choi
            Reporter:
            hyunsik Hyunsik Choi
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development