Uploaded image for project: 'Tajo'
  1. Tajo
  2. TAJO-1408

Make IntermediateEntryProto more compact

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Trivial
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.11.0
    • Component/s: Data Shuffle
    • Labels:
      None

      Description

      List<Pair<Long, Pair<Integer, Integer>>> can be just a long[]

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user navis opened a pull request:

          https://github.com/apache/tajo/pull/428

          TAJO-1408 Make IntermediateEntryProto more compact

          Passed all tests

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/navis/tajo TAJO-1408

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tajo/pull/428.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #428


          commit 3a3b83edb6afad744cf0b523cc51051b0eeed39c
          Author: navis.ryu <navis@apache.org>
          Date: 2015-03-15T06:35:53Z

          TAJO-1408 Make IntermediateEntryProto more compact


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user navis opened a pull request: https://github.com/apache/tajo/pull/428 TAJO-1408 Make IntermediateEntryProto more compact Passed all tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/navis/tajo TAJO-1408 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tajo/pull/428.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #428 commit 3a3b83edb6afad744cf0b523cc51051b0eeed39c Author: navis.ryu <navis@apache.org> Date: 2015-03-15T06:35:53Z TAJO-1408 Make IntermediateEntryProto more compact
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jihoonson commented on the pull request:

          https://github.com/apache/tajo/pull/428#issuecomment-86778205

          I'll review today.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jihoonson commented on the pull request: https://github.com/apache/tajo/pull/428#issuecomment-86778205 I'll review today.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user jinossy commented on the pull request:

          https://github.com/apache/tajo/pull/428#issuecomment-87515265

          @navis
          It would be great if you share the improved compaction results.

          Show
          githubbot ASF GitHub Bot added a comment - Github user jinossy commented on the pull request: https://github.com/apache/tajo/pull/428#issuecomment-87515265 @navis It would be great if you share the improved compaction results.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user hyunsik commented on the pull request:

          https://github.com/apache/tajo/pull/428#issuecomment-99620435

          Hi @navis,

          The patch looks good to me. The effect is obvious because it just changes ```List<Pair<Long,Long>>``` into a list of ```long``` values. Here is my +1.

          There is one trivial things. You changed host to hostAddress, still meaning only hostname or ip address. But, the field includes ```hostname:port```. If you agree, I'd like to change it to just ```address```.

          Show
          githubbot ASF GitHub Bot added a comment - Github user hyunsik commented on the pull request: https://github.com/apache/tajo/pull/428#issuecomment-99620435 Hi @navis, The patch looks good to me. The effect is obvious because it just changes ```List<Pair<Long,Long>>``` into a list of ```long``` values. Here is my +1. There is one trivial things. You changed host to hostAddress, still meaning only hostname or ip address. But, the field includes ```hostname:port```. If you agree, I'd like to change it to just ```address```.
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tajo/pull/428

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tajo/pull/428
          Hide
          hyunsik Hyunsik Choi added a comment -

          committed to master branch. Thank you for your contribution.

          Show
          hyunsik Hyunsik Choi added a comment - committed to master branch. Thank you for your contribution.
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #335 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/335/)
          TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik) (hyunsik: rev fab63900cf44b61d571fb9c2982285bb8b669702)

          • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java
          • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
          • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
          • CHANGES
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #335 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/335/ ) TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik) (hyunsik: rev fab63900cf44b61d571fb9c2982285bb8b669702) tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java CHANGES
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Tajo-master-build #695 (See https://builds.apache.org/job/Tajo-master-build/695/)
          TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik) (hyunsik: rev fab63900cf44b61d571fb9c2982285bb8b669702)

          • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • CHANGES
          • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java
          • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Tajo-master-build #695 (See https://builds.apache.org/job/Tajo-master-build/695/ ) TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik) (hyunsik: rev fab63900cf44b61d571fb9c2982285bb8b669702) tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java tajo-core/src/main/proto/TajoWorkerProtocol.proto CHANGES tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
          Hide
          hyunsik Hyunsik Choi added a comment -

          It causes data loss. So, I reverted this issue. After I fix the bug, I'll commit this patch with the bug fix to master branch later.

          Thanks!

          Show
          hyunsik Hyunsik Choi added a comment - It causes data loss. So, I reverted this issue. After I fix the bug, I'll commit this patch with the bug fix to master branch later. Thanks!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-CODEGEN-build #375 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/375/)
          Revert "TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik)" (hyunsik: rev 7c8477dd3ac15188353044981746ecfef87f2f02)

          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java
          • CHANGES
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java
          • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
          • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
          • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-CODEGEN-build #375 (See https://builds.apache.org/job/Tajo-master-CODEGEN-build/375/ ) Revert " TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik)" (hyunsik: rev 7c8477dd3ac15188353044981746ecfef87f2f02) tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java CHANGES tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Tajo-master-build #736 (See https://builds.apache.org/job/Tajo-master-build/736/)
          Revert "TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik)" (hyunsik: rev 7c8477dd3ac15188353044981746ecfef87f2f02)

          • tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java
          • tajo-core/src/main/proto/TajoWorkerProtocol.proto
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java
          • tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java
          • tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java
          • CHANGES
          • tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java
          • tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java
          • tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Tajo-master-build #736 (See https://builds.apache.org/job/Tajo-master-build/736/ ) Revert " TAJO-1408 Make IntermediateEntryProto more compact. (Contributed by navis, Committed by hyunsik)" (hyunsik: rev 7c8477dd3ac15188353044981746ecfef87f2f02) tajo-core/src/test/java/org/apache/tajo/master/TestRepartitioner.java tajo-core/src/main/proto/TajoWorkerProtocol.proto tajo-core/src/main/java/org/apache/tajo/querymaster/Repartitioner.java tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppender.java tajo-core/src/main/java/org/apache/tajo/worker/ExecutionBlockContext.java tajo-common/src/main/java/org/apache/tajo/util/NumberUtil.java CHANGES tajo-storage/tajo-storage-hdfs/src/main/java/org/apache/tajo/storage/HashShuffleAppenderManager.java tajo-core/src/test/java/org/apache/tajo/querymaster/TestIntermediateEntry.java tajo-core/src/main/java/org/apache/tajo/querymaster/Task.java

            People

            • Assignee:
              navis Navis
              Reporter:
              navis Navis
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development