Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.8.0
    • Component/s: Data Shuffle
    • Labels:
      None

      Description

      Currently, some intermediate file is text format, Tajo should change to binary format.

      • Support configurable storage type(RAW, CSV)
      1. TAJO-435_2.patch
        78 kB
        Jinho Kim
      2. TAJO-435.patch
        78 kB
        Jinho Kim

        Issue Links

          Activity

          Hide
          jhkim Jinho Kim added a comment -

          I've attached the patch.

          Show
          jhkim Jinho Kim added a comment - I've attached the patch.
          Hide
          hyunsik Hyunsik Choi added a comment -

          I'm reviewing this patch.

          Show
          hyunsik Hyunsik Choi added a comment - I'm reviewing this patch.
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1
          I'm very happy about this work. I have looked forward to seeing this work for very long time. The patch looks great. There is only a trivial comment. Before you commit this work, could you please remove unused variables, methods, and debug log in RawFile? The below log seems unnecessary.

          if (LOG.isDebugEnabled()) {
                  LOG.debug("RawFileAppender open:" + path);
                }
          

          Thanks!

          Show
          hyunsik Hyunsik Choi added a comment - +1 I'm very happy about this work. I have looked forward to seeing this work for very long time. The patch looks great. There is only a trivial comment. Before you commit this work, could you please remove unused variables, methods, and debug log in RawFile? The below log seems unnecessary. if (LOG.isDebugEnabled()) { LOG.debug( "RawFileAppender open:" + path); } Thanks!
          Hide
          jihoonson Jihoon Son added a comment -

          +1
          This patch looks great to me!

          Show
          jihoonson Jihoon Son added a comment - +1 This patch looks great to me!
          Hide
          sirpkt Keuntae Park added a comment -

          +1
          This is really great patch!!
          When I applied this patch to my cluster and tested TPC-H query 3 and 10, I got dramatically reduced intermediate stage execution time (about 33% reduction).
          Great job!!

          Show
          sirpkt Keuntae Park added a comment - +1 This is really great patch!! When I applied this patch to my cluster and tested TPC-H query 3 and 10, I got dramatically reduced intermediate stage execution time (about 33% reduction). Great job!!
          Hide
          jhkim Jinho Kim added a comment -

          Thank all guys for the review.
          I've updated the second patch that remove useless codes.

          Hyunsik,
          I've updated debug log. it can help for debugging

          Show
          jhkim Jinho Kim added a comment - Thank all guys for the review. I've updated the second patch that remove useless codes. Hyunsik, I've updated debug log. it can help for debugging
          Hide
          hyunsik Hyunsik Choi added a comment -

          +1 Ship it!

          Show
          hyunsik Hyunsik Choi added a comment - +1 Ship it!
          Hide
          jhkim Jinho Kim added a comment -

          I've just committed it.
          Thanks.

          Show
          jhkim Jinho Kim added a comment - I've just committed it. Thanks.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-trunk-postcommit #630 (See https://builds.apache.org/job/Tajo-trunk-postcommit/630/)
          TAJO-435: Improve intermediate file. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=f08724f9c8bb80d93db0d5517fe6351a9a4d6e63)

          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java
          • tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/RawFile.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java
          • tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java
          • CHANGES.txt
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java
          • tajo-core/tajo-core-storage/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PartitionedStoreExec.java
          • tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/IndexedStoreExec.java
          • tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-trunk-postcommit #630 (See https://builds.apache.org/job/Tajo-trunk-postcommit/630/ ) TAJO-435 : Improve intermediate file. (jinho) (jinossy: https://git-wip-us.apache.org/repos/asf?p=incubator-tajo.git&a=commit&h=f08724f9c8bb80d93db0d5517fe6351a9a4d6e63 ) tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/DataChannel.java tajo-core/tajo-core-storage/src/main/java/org/apache/tajo/storage/RawFile.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/worker/TestRangeRetrieverHandler.java tajo-core/tajo-core-backend/src/test/java/org/apache/tajo/engine/planner/physical/TestPhysicalPlanner.java CHANGES.txt tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/PhysicalPlannerImpl.java tajo-core/tajo-core-storage/src/test/java/org/apache/tajo/storage/index/TestBSTIndex.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/PartitionedStoreExec.java tajo-common/src/main/java/org/apache/tajo/conf/TajoConf.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/physical/IndexedStoreExec.java tajo-core/tajo-core-backend/src/main/java/org/apache/tajo/engine/planner/global/GlobalPlanner.java

            People

            • Assignee:
              jhkim Jinho Kim
              Reporter:
              jhkim Jinho Kim
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development